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STUDENT PERCEPTIONS OF 
INTERNSHIP EXPERIENCES 



Thomas P. Schambach 
Illinois State University 

Jim Dirks 

Illinois State University 



ABSTRACT 

Internships are often omitted from or put-off until late in some students' programs of study. Furthermore, some 
academics are reluctant to accept the academic legitimacy of applied work programs, thus cooperative education 
programs are not offered or encouraged in some curriculums. This study invited students finishing their internship 
experiences to reflect on the legitimacy of internships as a method to enhance their educational experience and to 
prepare students for careers as computing professionals. The student subjects in this research were all computing 
majors (Computer Science, Information Systems, Telecommunications). Results reported in this study are based on 
two data collection methods. Fixed format scaled responses from a small sample (N=70) were used to gather and 
analyze description statistics of graduating interns 'perceptions. In addition, open format responses were evaluated, 
coded and summarized from a corresponding sample. Student responses were strongly favorable concerning their 
internship experiences. Most students described the internship as a great experience that had a major impact on 
their learning and on their understanding of real world issues and environments. Respondents overwhelmingly 
recommended that other students get involved in cooperative education opportunities, preferably early within their 
major program of study. 



INTRODUCTION 

Businesses need a diverse, well-educated workforce in 
order to successfully adapt to rapidly changing 
technology-enabled practices. Thus, to ensure ongoing 
competitiveness organizations and nations must 
continually develop their human capital, understanding 
that the educational system represents the pipeline to 
enable future success. Ensuring that graduates acquire 
appropriate competencies before entering the workforce 
is a joint responsibility of studaits, educational 
institutions, and potential employers. 

Internships (Co-operative Education) further the social 
and economic development of society using education 
programs that expand the learning experience of 
students by combining academic studies with work 
experience. Although variations exist, internships 
typically involve paid, full-time, temporary employment 



in a structured setting where the student works in a job 
role related to their degree focus. Internships typically 
include at least one full-time term (quarter, semester) of 
employment and sometimes require two terms. Students 
receive academic credit for their internship and often 
have assignments requiring them to reflect on the 
parallels between workplace experiences and the 
concepts learned in coursework. This experiential 
learning approach entails a different yet complimentary 
learning process relative to traditional classroom based 
educational setting. While an internship experience has 
some relatively obvious advantages for the student, it 
also provides significant advantages to the sponsoring 
firm, to the educational institution, and to workforce 
preparedness of the nation at large. 

Large employers increasingly demand work experience 
when recruiting new hires, including new graduates. 
Cooperative education based industrial internship 
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programs provide students an opportunity to gain 
practical work experience in their area of professional 
study. Supported by theories of e?q)eriential learning, 
experiential education can increase the motivation of the 
learner, improve long-term retention, and lead to a 
greater sense of personal accomplishment (Palmer, 
1987; Cross, 1994). Furthermore, an internship 
demonstrates that the student is both informed and 
serious about his or her career direction. The significant 
impact of real-world experience is supported by the 
report that over 90% of recruiters say internship 
experience is viewed as an important factor in screening 
job applicants’ resumes (Wilson 1997). Internship 
programs potentially provide benefits to the student 
intern, to the sponsoring organization, and to the 
academic department facilitating the cooperative 
education relationship. 

POTENTIAL BENEFITS OF INTERNSHIPS 

As mentioned previously, cooperative education 
(internships) potentially benefits the student, the 
employing firm, the associated educational institution, 
and more generally the preparedness of the national 
workforce. Benefits to the participating student include 
increased motivation of the learner, improved long-term 
retention of material, and a greater sense of personal 
accomplishment. The intern receives an extended 
opportunity to apply previously learned conceptual 
knowledge in a real organizational setting. Such 
settings are generally more complex than can be 
simulated in classroom based exercises or case studies. 
This real-world encounter reinforces conceptual 
learning, makes it more visible, and emphasizes task 
importance. Moreover, woiking in organizational 
settings makes the intern more aware of the importance 
of ‘soft skills’ such as effective communication, social 
interaction, teamwork, and ability to problem-solve in 
environmentswhere defining the problem is amajor part 
of the overall job. Other benefits for the intern include 
clarification of career goals, awareness of organizational 
settings, clarification of valuable competencies, 
increased relevance of learning, establishing 
self-confidence, financial assistance for educational 
expenses, contacts with and previews of potential 
employers, exposure to working role models (potential 
mentors), and an increase in marketable job skills that 
often positively impact employability and starting 
salary. Importantly, internships grant an opportunity to 
verify career interest and desire to pursue a career in a 
given profession or specialty area; the opportunity to 
become more aware of the skills most valued by 



industry; the opportunity to select courses and steer their 
remaining course-work based on enhanced perspectives 
gained during their internships. In addition, student 
interns reinforce previous course-work and better 
understand course topics by framing the academic 
concepts in terms of analogies and perspectives gained 
during real work experiences. 

In return for their mentoring investment, sponsoring 
corporations obtain relatively inexpensive professional 
labor while concurrently conducting a pro-longed 
interview as a basis for hiring decisions. Moreover, a 
mutually successful internship experience enables an 
advantageous personal relationship that facilitates 
recruitment of the student following graduation (Tobias, 
1996). Importantly, intern employers engage a 
cost-effective, low-risk means of evaluating potential 
future employees. When hired, interns tend to 
demonstrate high retention based on realistic 
expectations. Moreover, in tight labor markets firms 
may experience few graduate applicants if they forego 
the chance to recruit early in the students’ career 
preparation, since many interns receive offersfrom their 
internship employer. Furthermore, the organization may 
gain new perspectives or practices based on new 
technologies and techniques that the student intern 
transfers from their university experiences. Often the 
students can offer new ideas (based on classroom 
exposure to new concepts or technologies) and new 
perspectives. Some internships result in ongoing joint 
research relationships between the firm, student, and 
sponsoring university, and provide organizations with 
opportunities to influence curriculum design and 
content. 

For the associated educational institution internships 
enable an additional route for communication with the 
business community. Such liaison enables feedback 
concerning the relevancy and quality of academic 
programs. Thus, academic departments’ might benefit 
by strengthening their relationship with industrial 
partners, while also obtaining feedback (via intern 
assessment) concerning the quality of their academic 
product (valued student/graduates). Moreover, feedback 
from both the sponsoring organization and the student 
interns provide insights to the skills being sought in 
industry. These insights assist the department’s ongoing 
efforts in curriculum development and refinement. Such 
feedback, through direct communication with 
employers, and intern feedback about competencies 
used enable educators to adjust course offerings and 
course content to meet evolving needs of society. 
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Furthermore, upon returning to the classroom interns 
often demonstrate more confidence and more 
responsibility, and through sharing their field experience 
help to enliven discussion of academic concepts via 
examples of practical application, thus helping the entire 
class to better envision the relevancy of conceptual 
topics. 

DEVELOPING THE RESEARCH FOCUS 

Although several sources indicate students gain value 
from the internship experience (Tobias 1996; Wilson 
1995), the evidence to support benefit claims is largely 
anecdotal . For example, the Association for Computing 
Machinery (ACM) career consultant, Jack Wilson, says 
it is extremely important in today’s business 
environment to show evidence of relevant work 
experience. When university graduates are competing 
for top jobs at top companies they should expect to face 
tough competition from other academically qualified 
candidates. According to Wilson, “when you are 
competing for employment with other great students 
from good schools, with good grades and skills, your 
relevant work experience can make a big difference.” 

While the benefits proposed seem relevant and realistic; 
it is not evident that everyone concludes the benefits are 
real or warrant educational consideration. Although 
encouraged by some academics, experiential learning 
has skeptics who question the ability of workplace 
activities to assist in achieving academic goals (Gore 
and Nelson, 1984); opponents believe experience alone 
doesn’t warrant the awarding of academic credit or 
academic resources (Whitaker, 1989). Lacking evidence 
of academic value, college faculty grant relatively 
limited commitment and support toward experiential 
learning programs, such as co-ops and internships (Gore 
and Nelson, 1984). Although experiential learning 
seems particularly relevant within applied disciplines 
(Cross, 1 994), there is a scarcity of research concerning 
the success of internships within the realm of 
information systems education. 

In the Applied Computer Science (ACS) programs at 
Illinois State University students are highly encouraged 
(semi-mandated) to participate in a relevant computing 
related internship. While a large preportion of ACS 
students seem eager to gain an internship experience, 
others are less enthusiastic, and some are skeptical about 
delaying their entrance into the real job (permanent 
employment) market, especially in an era (1998-2000) 
when jobs were plentiful. A few even voiced the 



opinion that the internship program is a university 
supported industrial conspiracy to co-opt talented 
professional labor at below market wage rates. 

RESEARCH QUESTIONS 

The purpose of the current study is to analyze data that 
empirically examines claims related to benefits incurred 
by student interns. The research questions being 
evaluated are summarized in Table 1. We anticipate 
these findings will be meaningful to future students, to 
curriculum planners, to the faculty supporting the 
internship program, and to our many industrial partners 
who continue to demonstrate interest in sponsoring 
internship contracts and hiring graduates with intemshp 
experience. 

METHODOLOGY 

Graduating internship students are used as the data 
source of this study in order to obtain credible results 
from the perspective of prospective internship students. 
Graduating internship students are defined here as 
students who have successfully completed an interndiip 
experience; however, these students normally have 
additional course-work to complete before they’ll 
graduate from ACS degree programs. The analysis of 
survey data was conducted in two ways. First we 
conducted an analysis of previous student comments 
concerning internship experiences. This included 
parsing, coding, categorizing, and summarizing student 
comments regarding their internship experiences. These 
comment categorizations were then used in conjunction 
with a literature review to construct a Likert-type scaled 
survey instrument. The closed-format survey used a 
five point scale ranging from 1 = Strongly Agree to 5 = 
Strongly Disagree. 

The scaled survey instrument was completed by a subset 
of Summer 1997 and Summer 1999 student interns. The 
scaled survey subset is composed of students who 
returned their standard internship evaluation forms 
during regular office hours and thus could be requested 
to complete this supplemental survey. Statistical results 
reported in this study are based on 70 scaled-survey 
responses along with a representative sample of student 
comments from a larger respondent sample who 
submitted a standard internship evaluation used by our 
cooperative education office. Open format narrative 
responses were culled from optional student replies to 
the question “What would you tell other young ACS 
majors trying to decide whether or not to become 
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TABLE 1 

RESEARCH QUESTIONS 



The following research questions are based on the perceptions of graduating internship students. 


1. 


To what extent did the internship provide an opportunity to gain real-world work experience that seems 
to be valued by industry/recruiters. 


2. 


To what extent did the internship provide you an opportunity to verify whether to pursue a career in the 
I/S profession. 


3. 


To what extent did the internship provide an opportunity to learn more about a potential employer. 


4. 


To what extent did the internship provided an opportunity to gain confidence in professional skills and 
capabilities. 


5. 


To what extent did the internship provide an opportunity to learn valuable skills that would be difficult 
to learn in a classroom. 


6. 


To what extent did the internship provide the background to better understand course- work by 
comparing course concepts to real world computing experience. 


7. 


To what extent did the internship provide an q>portunity to gain awareness of what skills are used and 
valued in the workplace. 


8. 


To what extent did the internship enable improved interpersonal communication skills. 


9. 


To what extent did the internship enable improved technical skills. 


10. 


To what extent did the internship provide an opportunity to redirect computing studies toward specific 
areas of interest. 


11. 


To what extent did the internship provide an opportunity to select future courses based on a more 
informed perspective. 


12. 


Overall, how enjoyable was the coop/intemship experience? 


13. 


Overall, how valuable was the coop/intemship experience? 


14. 


Financial compensation received was good (fair market value). 


15. 


Would you recommend that other students participate in a coq>/intemship experience? 



involved in the Cooperative Education Program?” 
Narrative responses were parsed into 145 comments that 
were then coded fcr classification. These open format 
narrative comments are used to add richness and to 
supplement the descriptive statistical analysis. 

RESULTS 

Student responses provide strong evidence that the 
internship experience isworthwhile and valuable. Table 
2 illustrates the percent of respondents who agreed or 
strongly agreed (item response=l), the mean score for 



the item on a five-point scale, and the standard 
deviation. The survey items’ number reflects the 
research question from Table 1. 

Responses to the first research question show near 
unanimous agreement that student interns perceive the 
experience gained to be valuable in terms of industry 
recruitment. Sixty-seven of seventy respondents believe 
the internship provided a valuable real-world 
experience. Eighteen parsed comments relative to this 
question included several declaring job offers, others 
noting prospects for future employment with their coop 
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TABLE 2 

DESCRIPTIVE STATISTICS REGARDING INTERNSHIP EXPERIENCE 



Survey Item 


% Agree or 
StronglyAgree 


Mean 

Score 


Standard 

Deviation 


Item I : valuable real-world experience 


95.7 


1.34 


.56 


Item 2: verify decision on I/S career path 


78.6 


1.80 


.93 


Item 3: learn more about a potential employer 


87.1 


1.66 


.90 


Item 4: gained confidence in my skills, capabilities 


88.6 


1.57 


.69 


Item 5: learned skills difficult to learn in classroom 


88.6 


1.49 


.81 


Item 6: background to better understand course-work 


84.3 


1.76 


.81 


Item 7: gained awareness of valued skills in workplace 


97.1 


1.53 


.56 


Item 8: improved my interpersonal skill 


90.0 


1.70 


.69 


Item 9: improved my technical skills 


87.1 


1.60 


.75 


Item 10: helped re-direct my studies to area of interest 


78.6 


1.77 


.89 


Item 1 1 : enables course selection from more informed basis 


68.6 


2.10 


1.01 


Item 12: Overall, how enjoyable was the coop experience 


95.7 


1.51 


.63 


Item 13: Overall, how valuable was the coop experience 


95.7 


1.46 


.63 


Item 14: Financial compensation received was good 
(fair market value) 


47.1 


2.34 


1.43 


Item 15: Recommend that other students participate in a 
coop/intemship experience (Yes/No item) 


97.1 







sponsor, and others declaring valuable experiences that 
would help build their resume. 

Response to the second research question shows strong 
agreement that the internship helped to verify their 
desire to pursue a career in the computing profession. 
While most respondents agreed (79%) only three 
indicated disagreement with the statement. Coop 
experience allows students to get a taste of the 
real-world and thus to verify whether their targeted 
career area is in fact something they really want to do. 
For example, “Coq? is a great experience — I got a taste 



of what my future job/career will be like” was reported 
by one student. In some cases real-world experience can 
also help confirm areas in which the person does not 
want to work. For example, one student commented “my 
coop helped me realize I don’t want a career in COBOL 
coding.” 

Item 3 three responses demonstrate widespread 
agreement (87%) that the internship provided the 
opportunity to learn more about a potential future 
employer. Limited response variance is reflected in the 
low standard deviations and seems to reflect widespread 
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respondent agreement concerning the benefits associated 
with their internship experience. Although the intern 
program is not intended to lead directly to job offers, 
several students reported via the free-format evaluation 
responses that they had accepted jobs with the internship 
sponsor. This correspaids with verbal comments 
received from many near-graduation seniors that they 
intend to start work with their internship sponsor. In 
addition to “firm” job offers for more senior students, 
several less advanced students commented on the 
internship as “a great way to get your foot-in-the-door 
with a good company” by building a network of 
business contacts. 

Responses to research question four and five 
demonstrate that approximately 89% of students 
perceived the internship experience to have increased 
their professional self-confidence while also providing 
an opportunity to learn valuable skills that would be 
difficult to learn in a classroom environment. Student 
comments indicate learning to work cooperatively in 
large project teams, and to gain awareness of business 
etiquette, politics, and ambiguity. In addition, students 
learned that they can be successful in the 
semi-structured context of the real world and that there 
is value to the skills and knowledge they have been 
gaining from coursework. For example, one student 
commented that “coop builds confidence in your 
abilities and the value of what your learning (in class).” 

Responses to research question six indicates 84.3% of 
the respondents believe their internship will improve 
and benefit their understanding of course concepts by 
providing real-world experience for comparison and 
analogy. Narrative responses note the internship was 
valuable not only for learning new things but also for 
reinforcing skills learned in the classroom. 

Responses to research question seven showed almost all 
students (97.1%) agree their internship gave them a 
better awareness of what skills were used and valued in 
the workplace. Narrative comments suggest benefits in 
viewing new technologies in use and witnessing 
emerging technological trends through the eyes of 
mentors and other professionals. 

Responses to research question eight and nine 
demonstrate that internship enabled students to enhance 
both their interpersonal communication (90%) and 
technical skills (87.1%). Many respondents commented 
that more learning occurs during the coop than in most 
classes. This tendency was especially salient in regard 



to soft skills, teamwork, and an appreciation for 
organizational environments and the realization of 
uncertainty. Importantly, only one student disagreed that 
their interpersonal communications skills were 
improved by their internship. “Soft skill” enhancement 
was also supported by narrative comments. 

Responses to research question ten and eleven display 
wide-ranging agreement concerning the value of 
internships to enable more informed direction and 
decisions concerning the targeting of personal academic 
programs and courses toward specific career interest 
areas. Nearly 79% of student respondents agree the 
coop experience helps in targeting programs of study 
while approximately 69% believe it will enable selection 
of courses from a more informed perspective. Analysis 
of free-form comments indicate that for some students 
the internship experience came too late in their degree 
program because their few remaining courses were 
pre-determined by degree requirements. Other 
comments noted that the internship experience 
reinforced existing course-of-study plans rather than 
enabling new decisions. For example, one student 
declared “I only became more convinced that I want to 
take a course involving the design and implementation 
of web pages.” Another student noted that “coop is a 
good idea; early in curriculum it can reshape your 
education.” Another advantage of an early curriculum 
internship experience is that it allows for multiple coop 
experiences in varied work settings. As noted by one 
aspiring coop enthusiast “Coop is a great experience — I 
plan to do it again!” 

Responses to research question twelve informed us that 
almost all (96%) respondents found their internship to 
be enjoyable. Only one respondent disagreed. 
Moreover, response to research question thirteen 
demonstrates the students’ collectively (96%) perceived 
value of participatiai in an internship assignment. 
Again, only one of seventy respondents disagreed that 
the internship experience was valuable. Student 
comments, such as, “the intern experiaice is invaluable” 
leave little doubt as to the perceived value of time and 
efforts devoted to the internship program. In the words 
of another student “Do it! Coop is a great experience, 
plus you get paid!” 

The only survey item receiving wide variability in 
response related to the fairness of compensation. 
Forty-seven percent rated compensation as “good” and 
nearly 39% rated compensation as “fair” while 
approximately 14% rated the compensation as poor. 
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Given that some students delay their internship until the 
final semester of their senior year it is understandable 
that some senior students would view internships as a 
delay in their procuring a real-job at real-job 
compensation rates (which averaged about $40,000 per 
year for new IS graduates at the time of the study). 
Nonetheless, practical experience is required for 
graduation and so all ACS students must participate in 
a full term internship or demonstrate an alternative 
method by which they gained practical IS work (project) 
experience. 

Responses to research question fifteen strongly support 
the wisdom of “Just Do It”. Item fourteen asked for a 
simple Yes/No answer to the question “Would you 
recommend that other students participate in a 
coop/intemship experience?” Nearly unanimous (97%) 
agreement to this question indicates that intanship 
experienced students overwhelming support the merit of 
cooperative education programs. While many narrative 
responses encouraged future studenis to take advantage 
of internship opportunities, several stronger comments 
declared the coop experience “should be absolutely 
required.” 

DISCUSSION AND LIMITATIONS 

Strong agreement, such as demonstrated above for item 
fifteen, is potentially in-part an artifact of the limited 
sample used for analysis of scaled responses ona survey 
distributed by the department’s coop coordinator. 
Nonetheless, respondents were provided with the ‘cloak 
of anonymity’ and thus were under no personal threat or 
influence regarding their responses. Furthermore, the 
free-format responses received from a larger sample 
were also nearly unanimous in supporting the value of 
internship experiences. Even the few negative comments 
received did not contradict the learning or professional 
value of a coop experience. The few negative comments 
received focused on either paying fees for internship 
course credit, or a feeling that the student had been 
placed in a non-challenging or unpleasant work context. 
For example, one student protested “I don’t see why we 
are required to pay ISU for internship hours — I found 
the job myself and used no ISU resources!” In contrast 
to this complaint other students noted “it’s great that 
working got me school credit” and that the coop office 
made finding a placement easy. Another less 
enthusiastic student noted “it wasn’t the experience I 
was hoping for but now I have a better perspective on 
the real world.” This last, somewhat negative. 



commentary suggests the student had in fact actually 
valued (if not enjoyed) the internship experience. 

In summary, a vast majority scaled survey responses 
show coop graduates’ perceived multiple benefits and 
value in their internship experience. Furthermore, 
approximately 98% of 145 parsed comments were 
favorable regarding the internship program. Moreover, 
the fact that intern graduates were willing to voluntarily 
spend time preparing comments to share perceptions 
about their internship experience is further evidence of 
their enthusiastic interest level regarding the 
coop/intemship program. 

The following student comments capture the essence of 
perspectives gathered from student comments. From 
one pragmatic studait we heard “It’s a great deal. You 
get course credit, work experience, and you get paid.” 
A more learning focused comment proclaims, “the 
amount of knowledge you gain through a coop is 
incredible” while another proclaims “my work 
experience complemented my school studies, topics that 
were unclear in class suddenly made sense when I had 
the opportunity to work through the issues with a 
hands-on approach rather than just reading about them.” 
Furthermore, a student providing counsel for new 
computing majors stated, “I would definitely encourage 
other students to participate in an internship. The 
internship provided me with a real sense of how the 
business world operates and enabled me to apply the 
knowledge I learned in the classroom to a real world 
situation.” Finally, a senior student claimed the most 
pragmatic of benefits, “Great news! I will begin 
full-time employment with CAT in August.” 

In conclusion, students who have completed internship 
experiences highly recommend that other students invest 
in the opportunity to participate in cooperative education 
programs. The benefits perceived and reported by 
students include recruitment advantages, an excellent 
method of learning, better understanding of organi- 
zations and career focus, as well as reinforcement of 
course learned skills and enhanced confidence in their 
own professional capabilities. For institutions that are 
interested in serving the needs and values of their 
student stakeholder this research sends a clear message 
that students strongly value internship experiences. 
Institutions, faculty, curriculum planners, future 
students, and parents need to be aware of the very 
positive findings concerning participation in cooperative 
education workplace experiences. 
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ABSTRACT 

The growing importance of information technology and innovation in the market place brings with it a need for the 
better management of professional knowledge for knowledge workers in the New Economy. This paper illustrates 
some major steps underway at the University of New South Wales, Australia towards buildingan educational system 
for IS professionals that can meet the requirements of the Knowledge Economy. The paper describes new 
multidisciplinary curriculum initiatives and instructional learning developments considered or implemented at the 
School of Information Systems, Technology and Management. 



INTRODUCTION 

The changing economic landscape, particularly the 
growing importance of innovations for economic growth 
and competitive advantage in the New Economy, 
suggests the need for better management of professional 
knowledge of future knowledge workers in the field of 
IS. New-age workers are expected to be skilled at 
creating, acquiring and transferring knowledge and 
modifying their behaviour accordingly (Garvin, 1998). 
They are anticipated to be capable of continually 
expanding their capacity to create desired results, 
nurture new thinking patterns, set free collective 
aspirations and learn how to learn together (Senge, 
1 990). It is also suggested that inventing new knowledge 
should be a modem worker’s way of behaving or being 
(Nonaka, 1998). 

Organisations’ increasing demand for new skills and 
capabilities for future IS professionals necessitates a 
corresponding response from the Education sector. In 
general, these demands have not been adequately 
addressed (Seufert and Seufert, 1999). Major criticisms 
are directed at content that does not reflect the cross- 
disciplinary nature of the field, has no base in reality and 
does not cultivate creativity and problem solving skills 



and instructional methods that largely impart knowledge 
rather than allow constructing it through experience. 

The purpose of this paper is to illustrate some major 
steps made by the University of New South Wales, 
Australia towards building an educaticnal system for IS 
professionals that can meet the requirements of the 
Knowledge Economy. In particular, the paper will 
describe new multidisciplinary curriculum and 
instmctional/leaming developments considered or 
implemented at the School of Information Systems, 
Technology and Management. 

CONVERGING DISCPLINES 

During 1999 the School of Informatiai Systems and the 
School of Library, Archives and Information Studies at 
the University of New South Wales joined together to 
form one new school, the School of Information 
Systems, Technology and Management (SISTM).' 

The success of this merger is evident within the teaching 
programs of the School, which is now able to offer a 
program of study to students who can select from a 
range of Information Systems and Information 
Management courses. At Graduate level the School 
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offers a Masters of Commerce in both Information 
Systems and Information Management. At 
Undergraduate level it is possible to undertake a dual 
major in both Information Systems and Information 
Management. 

It should not be surprising to learn that many students 
are in fact taking up this opportunity and combining 
courses as diverse as Systems Analysis and Design (IS) 
with Information, Knowledge and Society {JM). Yet for 
many it will be surprising. Despite the inherent 
similarities between the disciplines of Information 
Systems and Librarianship until recent times there had 
been remarkably little interaction between the 
practitioners. 

With a ‘traditional’ focus on Management Information 
Systems (MIS) and formal computer based services that 
meet organizational and operational needs, the strategic 
significance of Information Systems to business has 
seen Information Systems aligned with faculties of 
Business, Commerce and Economics. Librarianship and 
information science, with its focus on the management 
of material artifacts, has traditionally been aligned with 
the Social Sciences and Arts faculties (Buckland, 1999). 

The international landscape shifted considerably in the 
late 1990s when Schools of both Information Systems 
and the Information Sciences began to change the focus 
of their curriculums to reflect requirements driven by 
the internet, networked telecommunications and the new 
paradigm of knowledge management (Davis et al., 
2001). At the School of Information Systems, 
Technology and Management accounting for change in 
the New Economy has resulted in the development of 
new programmes and new courses that reflect a 
convergence in the way we think about, and teach, 
digital technology, information content and business 
management. The case study at the conclusion of this 
paper reports on this change through the example of one 
course: IMGT2703 Electronic Records Systems. 

MULTIDISCIPLINARY 
CURRICULUM DEVELOPMENTS 

During 2000, a special working party, consisting of 
representatives from the School of Information Systems, 
Technology and Management and interested parties 
from other relevant university schools, developed a new 
stream of multidisciplinary graduate programmes of 
study. These programmes are articulated to provide a 
cross disciplinary perspective on E-Business Manage- 



ment, Knowledge Management and Services 
Management. The following sections presait specific 
details of each new programme as outlined in their 
respective proposal documents. 

E-Business Management 

The first special working party rqpori (Special 
Postgraduate Coursework Programs: E-Business 

Management, 2000) describes E-Business in terms of 
‘doing business electronically,’ and differently. It 
recognises that E-Business involves the use of electronic 
technologies and the internet together to design, 
produce, and deliver goods and services — at speed, and 
globally. It also notes that E-Business operates through 
new business models, where agility is the key to success 
and organisational strategies, p*ocesses, competencies 
and cultures are fundamentally changed. Another 
important aspect of E-Business mentioned is that it blurs 
the distinction between organisations, customers and 
suppliers in value chains, and alters the meaning of 
working in, or for a business. The report further 
identifies that E-Business involves the gaining and 
sustaining of competitive advantage against a back- 
ground of industry and institutional transformation — 
where competitive advantage is to be found through 
industry recreation, extending the ‘reach’ of 
organisations across space and time, the ongoing 
enrichment of customers, the application of economic 
principles in new ways, the development and use of new 
types of law, the use of new forms of cultural affiliation, 
the inventive application of technology, and new forms 
of management. 

Although the report clearly states that E-Business 
depends upon technological expertise, ingenuity, and 
innovation, it also warns that these competencies will be 
insufficient to drive business success. It suggests that 
business success will flow from the design, use and 
redesign of appropriate and effective business models 
built around the use of e-technologies, and their 
management in the midst of profound change. Thus, 
technology, business models, and managemait are 
implicated together in creating successful and sustained 
E-Business outcomes. 

Following this discussion, the report proposes a series of 
Special Programs in E-Business Management that 
provide multi-disciplinary per^ectives on the 
conditions and drivers of E-Business success. They 
explore: alterations to work, organisations and 
management to accommodate and capitalise on E- 



10 



Proceedings of the 17^ Annual Conference of the International Academy for Information Management 



Business technologies; shifts in institutional frameworks 
which facilitate or condition E-Business — globalisation, 
communication and information technologies, and 
economic and legal institutions; transformations in the 
nature of change, in terms of speed, space, time, 
interdependencies, and heightened levels of ambiguity; 
trends, social impacts, long-term effects related to E- 
Business, and progressive shifts in the nature of E- 
Business itself. 

Knowledge Management 

The second special working party report (Special 
Postgraduate Coursework Programs: Knowledge 

Management, 2000) provides the following justification 
for introducing programmes of study in knowledge 
management. First, the report views knowledge 
management is an emergent response to a ‘third wave,’ 
digital or knowledge economy that is replacing the 
industrial society that has prevailed for the last two 
hundred years or so. Second, the rq)ort endorses 
Drucker’s (1993) argument that ‘in a knowledge 
economy the only thing that increasingly will matter in 
national as well as international economics is 
management’s performance in making knowledge 
productive.’ 

Furthermore, it recognises that organisations in the 
knowledge economy increasingly will inhabit 
environments that are chaotic — where the link between 
cause and effect becomes difficult to discern, small 
changes can be amplified beyond comprehension, and 
the future eludes prediction. The report notes that in this 
environment, organisations live with an inherent 
ambiguity, whilst competing on the edge of stability and 
instability. Only two things are believed to be certain 
for such organisations — their own decomposition as 
product/service life cycles rapidly change, and the 
impossibility of focusing organisational futures around 
known strategic portfolios. The report argues that 
survival depends on ceaseless innovation, and a 
capability to find opportunities for the exercise of new 
strategies. 

The report also argues that, in the knowledge economy, 
the intellect or knowledge of people will be the primary 
resource that is accumulated, developedand enhanced in 
the battle for competitive advantage. It predicts that 
success will accrue to organisations which can: offer 
ongoing, enriching service to individual customers, who 
perceive these offerings not as products but as solutions 
to their particular problems or needs; establish 



themselves as integral parts of diverse butrelevant value 
chains that permit the leveraging of resource use, whilst 
capitalising on their own distinctive capabilities; 
establish flexible, responsive, proactive — but 
directed — organisational processes that thrive on change 
and uncertainty, as means of exploiting market and 
competitive opportunities routinely and profitably; 
capitalise on knowledge resources available inside and 
outside the organisation, through the effective use of 
technology, diverse cultures, and modes of management 
that are visionary, change oriented, and inclusive; and 
mobilise a ‘strategic intelligence’ that is sufficient to 
sustain organisational identity and capabilities, whilst 
negotiating the ongoing, radical change driven by new 
service offerings. 

Finally, the report describes knowledge management as 
an organisational phenomenon that involves: apprecia- 
ting differences in types of knowledge and ways of 
knowing, and their personal, public and cultural 
manifestations; understanding the underlying economics 
of knowledge development and use, including the 
effects of rapid dissemination and the possibility of 
increasing returns on knowledge resources; accessing 
forms of legal right and remedy that protect proprietary 
or user advantages in knowledge resources; appreciating 
the nature of ‘knowledge work,’ and the needs and 
expectations of ‘knowledge workers’; appreciating 
relationships between knowledge, learning and 
innovation in, and by organisations; designing and 
negotiating systems for recognising and valuing the 
knowledge creation and utilisation capabilities of 
organisations; designing knowledge management 
architectures, systems and processes in organisations; 
impacting processes by which knowledge is mobilised, 
conserved, leveraged and enhanced within 
organisations; negotiating knowledge creation, diffusion 
and use within and across organisations and cultures, 
and in relationships with customers, suppliers and other 
stakeholders; managing knowledge strategically, in 
generating new service offerings and enhanced 
organisational capabilities. 

Accordingly, a series of Special Programs in Knowledge 
Management are proposed and designed to: provide 
multi-disciplinary perspectives on knowledge 
management as an emergent organisational 
phenomenon; provide an orientation to working and 
managing in contexts where knowledge is a central 
capability and a driver of organisational success; and 
provide choice in adapting study programs to academic 
or work backgrounds and career aspirations or needs. 
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Services Management 

The third working party report (Proposed Special 
Programs in Services Management, 200 1 ) deals with yet 
another emerging and ICT based economic 
development. According to this report, inmost Western 
economies, more than half of the Gross National 
Product is produced by the ‘services sector.’ In 
Australia, the ‘services sector’ accounts for more than 
sixty-five percent of Gross National Product, and 
employs close to seventy percent of the workforce; 
moreover, it is the fastest growing sector of the 
economy, and the most resilient during recessions. 

The report further notes that so pervasive and important 
is the services sector becoming that reference is now 
being made to a services economy as a new ‘economic 
order’ or business and management ‘paradigm’ that will 
characterise the early twenty first century. However, the 
report warns that these statistics or characterisations 
tend to understate the nature of the transformation that 
is taking place. It reveals that the distinction between 
goods and services is blurring, both in the minds of 
customers and in the strategic orientation of 
organisations. Most products sold not only embody an 
implicit service (how they will be used); they also are 
accompanied by a range of ancillary services — to the 
point where the provision of goods or products is seen 
simply as part of the provision of services to customers. 
In this sense, most firms are in the services economy, 
competing through theirdifferentiated service offerings. 
According to the report, services thus become the 
central focus of relationships between any organisation 
and its customers, and the central focus of organisational 
strategies and operations. 

The report identifies that the critical issue for service 
oriented organisations is customer satisfaction and 
enrichment, as assessed in many ‘moments of truth.’ 
Consequently, it is being recognised that the capability 
of all those who come into contact with the customer 
directly or who affect customer experiences in manifold 
‘moments of truth’ is critical to success. For most 
service oriented organisations this involves the entire 
workforce. The workforce needs to be empowered to 
represent the organisation in pursuing customer 
enrichment, and entrusted to secure customer 
satisfaction. It also needs to be enabled to pursue such 
outcomes, in terms of access to information and 
possession of requisite knowledge, skills and attitudes; 
such enablement will involve thinking as well as doing, 
ongoing learning, and capacities for innovation. 



The report also suggests that the workforce needs to be 
seen as the vehicle through which the guiding strategic 
intelligence of the organisation is deployed and realised, 
as it focuses its endeavours in best representing the 
organisation. Thus, the workforce needs to understand 
and embrace the strategic intelligence which guides the 
organisation. Location of the workforce within 
facilitative structures, work processes and cultures, so 
that it is capable of generating customer enrichment on 
an ongoing basis. The report recognises that it is 
becoming progressively virtual — a resource that alters 
in construction as service offerings vary, and which is 
increasingly casualised, contract based and mobile. 

With respect to managing service oriented organisations 
the report calls for a new role for management, together 
with revised approaches to managing. The key role for 
management will be to leverage the value of the 
organisation, by creating and re-creating (with all that 
this entails): A service-orientation; A capable 

workforce; A guiding form of strategic intelligence; An 
organisation that is both virtual and agile. This will 
require new approaches to individual, organisational, 
and inter-organisational development — as well as new 
approaches to structuring work, securing commitment, 
and retaining control. 

Consistent with the above discussion. The Special 
Programs in Services Management are proposed and 
designed to: provide multi-disciplinary perspectives on 
Services Management as a social and commercial 
phenomenon; provide an orientation to working and 
managing in service oriented environments; and provide 
choice in adapting study programs to academic or work 
backgrounds and career aspirations or needs. 

INNOVATIVE TEACHING AND LEARNING 

One of the major criticisms directed at cun*ent IS 
education is that a large amount of knowledge is 
imparted to the learner. Another noticeable weakness 
lies in the neglect of process oriented learning, that is, 
making the learning and thought process visible in order 
to develop the learners’ metacognition (Joyce and Weil, 
1986). There is a call for better balance between the 
imparting of knowledge to the learner and the learner’s 
own construction of it. A suggestion is made that the 
quantity of material to be learnt by telling should be 
reduced to a minimum and that the lesson time should 
instead be devoted to the cultivation of such qualities as 
problem-solving, decision making and creativity through 
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self-directed and collaborative learning. The 
complexities of learning and the large number of 
interacting factors which affect individual and group 
learning present many challenges. The following 
sections provide an overview of the latest innovative 
approaches proposed and considered for the use at the 
UNSW’s School of Information Systems, Technology 
and Management in a series of discussion documents. 

Technology-Mediated Teaching and Learning 

The Discussion Paper on Technology-Mediated 
Teaching and Learning (2001) states that the design of 
quality learning draws on the full range of digital and 
analogue media for its purposes. Currently, the Internet 
and other networked technologies attract the most 
interest. The document looks to technology to provide 
mechanisms and media to support learning strategies in 
three main modes: Adjunct — in which the technology 
supplements a course of study offered principally face- 
to-face; Mixed — in which technology partly replaces 
elements of traditional class interaction; and Online — in 
which all the content and processes of interaction are 
supported by technology. 

The document identifies the following as desirable ways 
in which online technology is used for the three modes 
of learning described. Access to a well- structured 
knowledge base: Using universal Internet standards, the 
student can access quality learning materials on demand, 
which are superior to those available or manageable in 
face-to-face settings; Active engagement with content: 
On the Internet, this is supported through the setting of 
tasks that may be published to the group or privately to 
the teacher via email or a student website. Other active 
engagement may be achieved through the development 
of pre-programmed interactive components or simula- 
tions that are made accessible through computer labs, 
face-to-face classrooms or online; Interaction with the 
teacher: Online discussion and dialogue may be held 
both synchronously through ‘chat’ sessions or more 
commonly through asyndironous discussions and 
bulletin boards. In a fully online course, this is the 
principal channel of communication within the group, 
but also in mixed and adjunct modes, it can facilitate 
interaction over and above that possible within the 
constraints of a face-to-face class. Opportunities for 
interaction with other learners: The online discussion 
group also enables student-to-student interaction that 
may be informal and initiated by the learners or a formal 
group task set by the teacher; Individual reflection on 
learning: Online learning incorporates explicit instances 



for reflection and reporting on cognition. Teachers also 
require ways to look back on the learning process and 
adjust strategies and activities to redress misconcep- 
tions; Feedback and formative assessment: Online 
groups offer an achievable and retrievable record of 
class interactions as a forum for formative feedback. 
Individual and confidential feedback may be provided 
via private threads or email. 

Furthemore, the document recommends that these 
processes within the School should be enabled through 
Support for development and adequate infrastructure. 
For teachers, the recommended support is in the form 
of: Guidelines regarding minimum standards for course 
development, design, and delivery; Processes and 
criteria by which learning materials are to be reviewed; 
Workshops and individual support in educational design 
in response to the specific needs of courses offered 
within a discipline. Technical assistance for staff in 
content design and interactive media production; 
Assistance in the transition from classroom to online 
learning processes, and in the development of adjunct 
materials through the modeling of best practice; 
Provision of feedback, in the form of reports from the 
learning system, on issues arising from student use of 
mediated learning material; Opportunities to share and 
discuss practice within the School through the formation 
of a learning community. For Students, the recom- 
mended support is in the form of: Specific information 
about the mediated learning and its use in each course, 
in addition to the basic Course Outline requirements; 
Training and information to equitably access courseware 
and other UNSW online resources; Access to technical 
assistance throughout the duration of a course/program; 
Prompt and accurate response to inquiries, which will be 
logged and recorded for later analysis; Opportunities to 
evaluate and comment on the teaching and learning 
process throughout a course. 

The document also suggests that the School needs to 
provide necessary infrastructure for the mediated 
teaching and learning purposes. This infrastructure 
includes: a technology plan that includes electronic 
security measures (i.e., password protection, encryption, 
back-up) to ensure quality standards and the integrity 
and validity of information held within the online 
teaching system; a local intranet, accessible to staff and 
students to enable high-speed access to email, 
courseware servers and teaching spaces; high-capacity 
local servers, to store and structure repositories of media 
content for courses; high speed connection with the 
external internet, to enable access to the resources and 
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connectivity of this global resource; modem connections 
for students to access School courses and course 
repositories from off campus; access to, and support for 
the software necessary to enable discipline discourse for 
both on and off campus groups; technical support for 
commonly used computer platforms and software, 
which is reviewed annually in the light of new 
technologies; templates for commonly used educational 
strategies, student and course web pages to minimise 
preparation time. 

Interactive Teaching and Learning 

The promotion of interactive teaching and learning 
within the School by The Discussion Paper on 
Interactive Teaching andLeaming (2001 ) reflects recent 
research into student learning, serves to build a 
community of practice which values and accommodates 
student diversity in learning, and is likely to improve the 
quality of learning experiences and satisfaction with the 
outcomes. Both staff and students are supported in 
various ways, to make the most of opportunities for 
interactive learning and to develop their own skills in 
interaction. A commitment to moving towards more 
interactive teaching practices is expected to enhance the 
quality of learning in the School. 

In this discussion paper, learning is recognised not only 
a process of cognitive development for the individual 
but also a social process of engagement with others 
within the learning environment. Recent constructivist 
theories of learning, place the individual as an active 
participant in her/his environment, rather than a passive 
recipient of stimuli. Learning is seen as a process in 
which the student constructs new knowledge, skills and 
understandings in response to her/his environment, 
continually integrating new experiences and information 
into existing cognitive structures and ultimately 
transferring that knowledge to new situations. The 
emphasis is on the processes of learning, rather than 
curriculum content. Principles of so-called deep learning 
also encourage active engagement with both content and 
other learners, along with opportunities to reflect on, and 
consolidate new knowledge into an existing knowledge 
base. The document supports Vygotsky’s description of 
the ideal as a ‘zone of proximal development’ in which 
the individual learner can continually expand know- 
ledge, skills and talents within a supportive framework 
or scaffold provided by the teacher and institution. 
Eventually, the ideal graduate becomes an independent 
learner who can maintain the process of knowledge 
construction outside of the safety of the institution. 



Furthermore, the document suggests three main reasons 
to teach interactively. First, interactive teaching offers 
some insight into what students actually know. This is 
its summative function, as it leads to testing and 
measuring student knowledge and understandings 
through questions, tests and exams. Second, interactive 
teaching is formative. The teacher seeks to direct 
students' cognitive processing along particular paths 
through conversations or dialogue. The resulting 
cognitive experience of the students will move them 
towards accepted conceptions of the topic within the 
discipline. Third, interactive teaching is motivational. A 
teacher has a responsibility to keep students interested, 
and this is more easily done when the student is actively 
involved. When teachers ask students to work in small 
groups on a case study or problem, the resulting 
discussion not only serves to build new knowledge, it 
also serves to motivate students. The anticipation of 
feedback from their peers or the teacher is a strong 
incentive. Interactive teaching methods can address each 
of these issues. Through well-designed learning 
processes, new material can be integrated into a 
student’s existing set of knowledge constructs in a way 
that provides for a deeper level of understanding to 
occur. 

The document notes that the following five skill-sets of 
teachers are seen to be associated with effective 
interactive learning by students: Using and Developing 
Professional Knowledge and Values; Communicating, 
Interacting and Working with Students and Others; 
Planning and Managing the Teaching and Learning 
Process; Monitoring and Assessing Student Progress and 
Learning Outcomes; and Reflecting, Evaluating and 
Planning for Continuous Improvement. In order to 
promote, maintain or develop these skill-sets by its 
academic staff, the School provides: opportunities for 
staff to discuss and evaluate interactive teaching; regular 
dissemination of current developments related to 
interactive teaching of disciplines of the School; 
technical resources, teaching spaces and infrastructure 
necessary for the conduct of a variety of desirable 
modes of active and interactive learning; staff 
development activities, such as workshops, seminars and 
individual coaching to build these skill-sets; time release 
for the planning and coordination of interactive teaching 
strategies across Courses and Programs. 

In addition, the document recognises that students also 
need specific competencies, guidance and support if 
they are to maximise their opportunities to learn 
interactively. Such opportunities are enhanced by: 
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Explicit statements by the School of expectations for 
student participation in learning. These statements 
should be communicated in advance of study through 
orientation and induction programs, and ideally should 
become an sign of the learning culture of the School; 
Skills in written and spoken communication sufficient 
for active learning by individuals and positive 
contributions to the learning of others; A level of 
metacognition by students regarding their own learning 
styles and preferences, and an appreciation of the role 
that culture and upbringing play in determining 
cognitive frameworks and learning; Self-management 
skills in identifying goals, setting priorities and 
independently managing time and resources towards 
meeting the expectations of a course; Sufficient levelsof 
technological literacy to access the learning materials 
and processes offered by staff. Support mechanisms, 
both educational and social, to address skills deficits or 
other impediments to participation and learning; 
Mechanisms by which students can provide evaluative 
feedback to staff in order to improve the design of 
learning processes. 

Cross-Cultural Teaching and Learning 

The Discussion P^er on Cross-Cultural Teaching and 
Learning (2001) reflects the commitment of staff and 
students to the development of effective cross-cultural 
learning in the school. This document is part of an 
ongoing discourse about the kinds of students we have 
enrolled in our School and the ways in which we wish 
approach their education. 

The document defines culture broadly as a set of values 
and beliefs shared by a group of people. Membership of 
such groups may be determined by birth, by choice or 
life circumstances. Cultural values and beliefs may be 
anchored by ethnicity, gender, religion, nationality and 
language. Students and staff in the School of School of 
Information Systems, Technology and Management 
have a range of cultural backgrounds and affiliations. In 
particular, the cultural diversity of students is striking. 
This diversity poses a number of critical issues for 
teaching and learning in the School. 

In order to help both teachers and students become more 
culturally aware of themselves and others, and to 
manage cultural diversity in the School, the document 
proposes the following eight principles: 



1. Be Consistent. We need to start by creating an 
environment in which the lules of interaction are 
apparent. Rather than trying to second-guess the 
competing expectations or prejudices of all, create 
a new ‘culture of the classroom' as a model for 
managing diversity and use this to mirror the global 
workplace. People respond to clear direction, 
especially in socially and culturally sensitive 
contexts or in processes in which they are unsure. A 
teacher may spend some time during the 
introductory weeks of a course negotiating what is 
expected and acceptable to the group regarding 
interaction, group work, questions andrespect. Staff 
and students should be confident that the general 
expectations are consistent across Programs and 
reflect those of the professional world. 

2. Provide Information. Wherever possible, be aware 
that miscommunication is the greatest impediment 
to learning and seek to provide information to all in 
accessible ways. Course content and assessment 
details, for instance, can be communicated in the 
Course Outline, on a website, in class and in 
individual consultations. Especially in the begin- 
ning, take the time to ensure that students have time 
to become familiar with the class culture and its 
expectations, and provide information in mult^le 
forms. 

3. Encourage Communication. As communication is 
the essential process whereby learning occurs, foster 
opportunities for students to express themselves. 
This will involve the use of questioning, discussion, 
debate class presentations and open invitations to 
contribute personal experience to cases studies. In 
class, allow ‘wait time’ in all interactions to 
encourage some individuals to overcome their 
desire to avoid participation. Ask if students know 
of other ways of approaching issues. Confirm and 
validate contributions with recognition and thanks. 
Give notice of a request for partic^ation — don’t 
spring a surprise on an unprepared student. 

4. Avoid Stereotyping. Stereotyping is how novice 
learners first sort and process different phenomena. 
They create large, easily managed categories that 
make sense of unfamiliar information. However, as 
expertise grows, learners modify these categories to 
differentiate the detail found in individuals. 
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Eventually, we become intellectually aware of 
individuals differences and can appreciate the 
dangers of generalisation. If stereotypes are used, 
for instance, in case studies, recognise this and 
explain why. Look beyond immediate physical and 
language differences to seek understandings of 
intention. 

5. Avoid Ethnocentrism. Appreciate that there are 
many views of the world. Avoid deficit models in 
which we suggest that other cultures simply lack 
some qualities that we value. Asian students 
consistently display higher scores on deep 
approaches and lower scores on surface approaches 
to learning despite the conception that they want to 
‘rote’ learn content. Promote equitable participation 
by all, rather than dominance by a few. 

6. Involve Others in Your Development. To see 
ourselves as others do is difficult. Check your 
perceptions with colleagues, and invite peer review 
of your style of teaching and interactiai. Ask others 
to help monitor your language and interpersonal 



dynamics. Ask peers to suggest and share 
techniques for motivating classroom interaction. 

7. Be an Example. Model inclusive language and 
behaviour where possible. Carry this through to the 
handouts, notes and OHTs used in class. Also, admit 
to uncertainties and ask others to suggest strategies 
in difficult situations. Seek to use global examples 
and analogies when illustrating a point. Avoid 
jargon or colloquialisms and model active listening. 

8. Structure Group Work to Manage Diversity. Make 
the team dynamic and its management a part of the 
assessment. Allow members to contribute in writing 
as well as verbally. Provide planning sessions to 
allocate responsibilities and follow-ip with support 
Make sure each member can access the information 
necessary for their component of the task— some 
overseas students do not have the same networks 
and resources as local students. Don’t force 
representational membership of groups, allow 
students with a common culture to work together. 



CASE STUDY 



IMGT2703: Electronic Record Systems 
Background 

When the School of Information, Library and Archives study joined with Information Systems in 1999 it brought 
with it the core subjects of an established programme in Information Management. The programme is acaedited with 
the Australian Library and Information Association, Australia’s profession organization for the library and 
information services sector. Among the core courses was provision fora course in Organisational Recordkeeping: 
Rights & Responsibilities. This course had previously been taught as part of an Archives and Recordkeeping 
programme which has not been continued within the new School. 

Although the Archives and Recordkeeping programme was not continued the importance of electronic records, 
document management and digital content management in networked information systems was widely accepted and 
provision made in 2001 for the continuation of the course. 

During 2000-2001, the School reviewednational and international curriculum developments in the field of electronic 
records and document management. Theresult of the review was the development of the course as it now stands. The 
current curriculum was taught for the first time during Second Semester 2001 and it is being taught again for Second 
Semester 2002. 

Course Description 

IMGT2703 provides an introduction to best practice in the management of electronic records. Issues that impact on 
electronic records management are the focus of the course. These include legal requirements, accountability and the 
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CASE STUDY 
(continued) 



role record management systems play in managing documents as strategic resources. The course covers current 
approaches concerning electronic records management, the records continuum and strategic organisational planning 
for information systems infrastructure. 

It examines methodologies and technologies for managing electronic records within the context of networked content 
management. It addresses the role of Metadata, Extensible Markup Language and Electronic Document Management 
Systems in managing recorded information as corporate assets. 

Cohort 

The course has demonstrated a broad appeal to undergraduate students from diverse fields of study. Enrolments for 
the first two years have included students from Accountancy, Cbmputer Scienceand Software Engineering, with the 
majority of students attending the course enrolled in the double major of Information Systems and Information 
Management. Although numbers are still small, with a total enrolment 20 students for Second Semester 2002, what 
has been surprising is that over a quarter of these students are enrolled in Computer Science or Software Engineering 
Degrees. 

£ Business Management 

The course empahsises an awareness of the impact of technologies on documenting business activities. A focus is 
placed on corporate accountability and the legal requirements for evidence of business transactions. Understanding 
the relationship of these requirements to the design and implementation of electronic record systems is a key 
objective of the course. Document design and metadata requirements for recording online transactions are examined. 

Knowledge Management 

The focus of the course is on managing information as acorporate asset. Indexing, metadata and information retrieval 
concepts from archival science and librarianship provide a foundation for managing records as an information 
resource. The role records play in corporate memory and organisational knowledge management are examined. The 
cultural, community and social contexts of recorded information are also investigated and students show a keen 
interest in exploring topics in this area, e.g. vital records and records of national significance, freedom of information 
& privacy, corporate accountability, as part of their research project. 

Service Management 

The course addresses recommendations proposed by Special programs in Services Management, focusing on digital 
library service models and digital content management models for publishing and managing electronic documents. 
Requirements for managing electronic records systems have structural similarity to the digital library and content 
management service models. 

Innovative Teaching & Learning 

The appeal of the course to computer science and software enginemng students was not anticipated. During 
discussions with the class, students have indicated that this appeal is derived from two aspects of the course: 

1. An emphasis on an analysis of the digital object itself (electronic records and documents) 



Proceedings of the 17'' Annual Conference of the International Academy for Information Management 



11 



CASE STUDY 
(continued) 



2. I The course’s emphasis on the use of information technologies in instruction 

Document Design and Metadata Laboratories 

Using the Extreme programming model ofprogramming partners as its reference frame, students ‘team-up’ in pairs 
for the course’s computer laboratories. During the labs they follow self-directed tutorials designed to introduce 
SGML/XML and metadata. The labs focus on concepts of logical document design and descriptive metadata and are 
supported by group sessions (a design workshop) where the relationship of the document and its internal structure 
to information systems is modeled and discussed. During the second half of semester students gain experience with 
TRIM, a commercial electronic document management system. 

Online Discussion Forum 

This year, for the first time, an online discussion forum has been added to the course. The discussion forum, vAiich 
is structured around formal coursework has been an extremely successful interactive learning activity. 

During the semester students are presented with three discussion topics. Each forum is structured around a problem 
statement or an issue derived from a course reading. 

After reading the article for a topic students select to either champion or challenge the discussion topic. Each topic 
continues for a period of three weeks during which time each student is required to make at least two contributions 
to the forum. Most students are posting at least three comments to each forum with many students posting four or 
five times on a topic. 

Although students are encouraged to keep postings informal and brief the average posting tends to be about 1 50-200 
words in length. Many postings include references to readings that students identify through their own research and 
a healthy debate around new and unexpected points often ensues. 

The course has a large number of students for whom English is a second language; for these students the forum is 
proving to be a successful cross-cultural leaning environment. All students are encouraged to express themselves in 
natural language: text messaging abbreviations (SMS style) and emoticons are welcome. 

As time progresses students are beginning to use the forum to discuss the course in a wider context and are using the 
space to help each other with research. 



ENDNOTE 

1 . The merger is not unique; other examples include the 
formation of the School of Information Management and 
Systems at Monash University, Victoria, Australia & the 
Graduate School of Information Management and 
Systems^ UCLA, Berkeley. 
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OF REQUISITE SKILLS NEEDED BY 
DEVELOPERS OF E-COMMERCE SYSTEMS 
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ABSTRACT 

The aim of this exploratory study is to identify the skills needed by developers of Electronic Commerce (e-Commerce) 
systems. The paper proposes a framework pertaining to three categories of e-Commerce development knowledge: 
technical, human, and organizational. The initial findings reveal that there are some 16 possible e-Commerce skills 
tapping the three areas of e^Commerce knowledge. The results also reveal that there are similarities and differences 
among the perceptions of Information Technology managers and researchers with regard to (1) the relative 
usefulness of the 16 skills, and (2) the extent to which new graduates of Information Technology programs are 
equipped with these skills. 



INTRODUCTION AND BACKGROUND 

No academic institution would argue against the fact 
that there is a compelling need to produce graduates 
with sufficient Electronic Commerce (e-Commerce) 
skills and knowledge to meet today's market needs. The 
demand of public and private organizations worldwide 
for professionals with appropriate e-Commerce skills is 
currently in the increase. e-Commerce development 
expertise now ranks as the most sought-after skill in the 
Information Technology community, according to a 
recent Computa^orld survey (Dash 2000). 

Despite all of this, to date little research had been 
carried out to address the issue of e-Commerce skills. 
Most of available investigations have been carried out in 
a non e-Commerce context. Ang and Jiwahhasuchin 
(1998), for example, discussed traditional Information 
Technology education in Thailand. Tafti and Shirani 
(1997) examined a model of traditional hierarchy of 
needs in an end-user-computing environment consisting 
of users' information-, system-, and technical support- 
related needs. Gupta et al. (1994) developed two 
questionnaires to assess a set of specific Information 
Technology areas and skills, university-industry 
collaboration activities, and training requirements for 



Information Technology graduates. Trauth et al. (1993) 
conducted a study to identify the key skills and 
knowledge areas that will be required of future 
Information Technology professionals. Nelson (1991) 
performed an assessment of educational needs that 
examined the deficiencies of both Information 
Technology and end-user personnel. All these studies 
were conducted in a non e-Commerce environment. 

The objective of this study is to fill part of this gap in 
the Information Technology literature. To achieve this 
goal, our paper, based on the work of Nelson (1991), 
Lee et al. (1995), Trauth et al. (1993), and Rada 
(1999), proposes a three-dimensional proficiency 
space for identifying the requisite skills needed by 
developers of e-Commerce systems that covers three 
sets of knowledge areas: technical, human, and 
organizational. 

INSTRUMENT DEVELOPMENT, 

DATA COLLECTION, AND RESULTS 

To develop the research instrument, we first interviewed 
the Information Technology manager (or a representa- 
tive) in 7 prominent private and public Kuwaiti 
organizations. During the interviews we asked them to 
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discuss the basic areas of knowledge needed by 
developers of e-Commerce systems and to highlight 
important skills within each area of knowledge. The 
interviews revealed several important high-level skills 
pertinent to the development of e-Commerce systems. 
Later, we organized these high-level skills in one list, 
deleted duplicate items from the list, and supplemented 
the list with certain of the high-level skills that we 
identified during our literature review. We pilot tested 
the instrument with two Information Technology 
managers and two Information Technology researchers. 
The managers and researchers provided minor albeit 
useful comments, which we incorporated in our final 
questionnaire. The final version of the questionnaire 
consists of two sections. The first section collects 
information on the respondents' background such as 
gender, age, education, etc. The second section requires 
the respondents to rate the usefulness of the 16 high- 
level skills, categcrized into three knowledge areas. It 
also requires the respondents to indicate the extent to 
which newcomers who recently graduated from an 
academic institution are competent in these skills. All 
questions were anchored around a five-point scale, 1 = 
very low usefulness/competency, 2 = low usefulness/ 
competency, 3 = medium usefulness/competency, 4 = 
high usefulness/competency, and 5 = very high 
usefulness/competency. 

Rada ( 1 999) maintains that the effectiveness of a worker 
can be characterized by how well he/she performs a list 
of individual tasks. Table 1 shows examples of high- 
level skills needed by developers of e-Commerce 
systems. The first set of skills relevant to technical 
knowledge and includes 6 high-level skills necessary for 
physical development of e-Commerce systems such as 
Web programming, Web internetworking, Web security, 
and the like. The second set of skills relevant to human 
knowledge and embodies 5 high-level skills such as 
interpersonal communication, problemsolving, conflict 
resolution, and so on and so forth. The last set of skills 
relevant to organizational knowledge and covers 5 high- 
level skills such as the ability to facilitate the integration 
between the e-Commerce system with the organization 
and existing information technologies. 

The questionnaire was sent to the Information 
Technology manager (or a rq)resentative) in 15 private 
and public organizations operating in Kuwait and to 15 
Information Technology academicians/consultants. All 
questionnaires that were sent were completed and 
returned to the author. Table 2 summarizes usefulness 
rankings of high-level skills needed by developers of e- 



TABLE 1 

HIGH-LEVEL SKILLS NEEDED BY 
DEVELOPERS OF E-COMMERCE SYSTEMS 



Knowledge 


High-level 


Area 


Skill 


Technical 


Web programming 
Web networking 
Web databases 
Web security 
Web management 
Web site design 
Interpersonal communication 
Problem solving 


Human 


Conflict resolution 




Collaboration 
Dealing with change 
Organizational goals and objectives 
Organizational policies and 
procedures 


Organizational 


Organizational functions and 
processes 

Organizational culture 
Organizational constraints 



Commerce systems as seen by Information Technology 
managers and academicians. It is evidoit from the table 
that there are some agreements and disagreements 
among Information Technology managers and academi- 
cians with regard to usefulness of the different skills 
needed for developers of e-Commerce systems. For 
example, both groups of Information Technology 
managers and academicians agree that web security, 
networking and programming skills are important for e- 
Commerce professionals. However, the same two 
groups disagree with regard to the importance of 
organizational goals and procedures. While Information 
Technology managers perceive organizational goals as 
less useful a skill to the work of e-Commerce 
developers. Information Technology academicians see 
the opposite. Information Technology academicians 
rank understanding the goals of the organization as the 
third most useful/important skill for work in an e- 
Commerce context. In the same vein, while Information 
Technology managers rank organizational procedures 
high in the usefulness scale. Information Technology 
academicians put the same skill last in the usefulness 
list. An interesting finding that comes out of our study 
is that Information Technology managers believe that 
skills relevant to the technical knowledge area (web 
security, web networking, web programming, web 
databases, web management, and web site design) are 
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the most useful skills needed by developers of e-Com- 
merce systems. In addition, Information Technology 
academicians to some extent share this view except the 
fact that they believe problem solving and understanding 
organizational goals and objectives are more useful than 
some technical skills such as web management. 

Table 3 shows Information Technology managers and 
academicians' ratings of the extent to whidi recently 
graduated Information Technology professionals are 
competent in e-Commerce skills. The table reveals that 
once again there are some similarities and differences 
among Information Technology managers and 
academicians perceptions with regard to perceived e- 
Commerce skills and knowledge of recently graduated 
Information Technology professionals. For instance, 
both groups of Infcrmation Technology managers and 
academicians agree that newly graduated Information 
Technology professionals are well trained in 
interpersonal communication, problem solving, 
collaboration, and dealing with change. Nonetheless, 
the perceptions of Information Technology managers 
and academicians differ with regard to the level of 
training that newly graduated Information Technology 
professionals receive on several skills, most notably the 
preparation they receive on web security, conflict 
resolution, web programming, web site design, 
understanding organizational goals, and following 
organizational procedures. Although Information Tech- 
nology managers perceive newly graduated Information 
Technology professionals as inadequately trained on 



web security, web programming, web site design and 
understanding organizational goals, Information 
Technology academicians believe their students are 
getting adequate training on these subjects. Moreover, 
whereas Information Technology managers perceive 
newly graduated Information Technology professionals 
as adequately trained on conflict resolution and 
organizational policies and procedures. Information 
Technology academicians believe their students are 
getting inadequate training on the same two skills. 
Another interesting finding that comes out of this study 
is that Information Technology managers believe that 
students usually come to the market prepared to exploit 
web databases. 

Table 4 summarizes the usefulness-competency gap as 
perceived by Information Technology managers. 
Overall, the results reported in Table 4 reveals that 
Information Technology managers believe that the 
competency level of new Information Technology 
graduates in all skills, but conflict resolution, is below 
what they believe could be useful for their current work. 
Web security, web programming, web management, 
web site design, and web networking come at the top of 
the discrepancy list as perceived by Information 
Technology managers. Besides that, these skills come 
at the top of the mismatch between the usefulness- 
competency gap as perceived by Information Techno- 
logy managers and the same gap as perceived by 
Information Technology academicians. 



TABLE 2 

USEFULNESS RANKINGS OF E-COMMERCE SKILLS— IT MANGERS VS. IT ACADEMICIANS 



High-level Skill 


Usefulness Rank 




IT Managers 


IT Academicians 


Web programming 


3 


4 


Web networking 


2 


2 


Web databases 


4 


7 


Web security 


1 


1 


Web management 


6 


9 


Web site design 


5 


6 


Interpersonal communication 


9 


8 


Problem solving 


7 


5 


Conflict resolution 


16 


15 


Collaboration 


13 


13 


Dealing with change 


11 


1 1 


Organizational goals and objectives 


12 


3 


Organizational policies and procedures 


8 


16 


Organizational functions and processes 


14 


10 


Organizational culture 


15 


12 


Organizational constraints 


10 


14 
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TABLE 3 

THE COMPETENCY OF NEW IT GRADUATES— IT MANGERS VS. IT ACADEMICIANS 



High-level Skill 


Competency Rank 




IT Managers IT Academicians 


Web programming 


14 


6 


Web networking 


6 


2 


Web databases 


1 


7 


Web security 


16 


4 


Web management 


15 


1 1 


Web site design 


13 


5 


Interpersonal communication 


8 


9 


Problem solving 


3 


3 


Conflict resolution 


4 


14 


Collaboration 


II 


10 


Dealing with change 


9 


12 


Organizational goals and objectives 


12 


1 


Organizational policies and procedures 


2 


15 


Organizational functions and processes 


5 


8 


Organizational culture 


10 


16 


Organizational constraints 


7 


13 


TABLE 4 




THE E-COMMERCE USEFULNESS-COMPETENCY GAP— IT MANGERS VS. IT ACADEMICIANS 


High-level Skill 


Usefulness-Competency Gap 


IT Managers IT Academicians 


Web programming 


1.93 


0.20 


Web networking 


1.53 


0.20 


Web databases 


0.67 


0.13 


Web security 


2.67 


0.33 


Web management 


1.60 


0.00 


Web site design 


1.53 


0.00 


Interpersonal communication 


0.60 


0.00 


Problem solving 


0.60 


0.00 


Conflict resolution 


-0.07 


-0.40 


Collaboration 


0.40 


-0.53 


Dealing with change 


0.47 


-0.20 


Organizational goals and objectives 


0.60 


0.00 


Organizational policies and procedures 


0.33 


-0.40 


Organizational functions and processes 


0.07 


-0.40 


Organizational culture 


0.20 


0.00 


Organizational constraints 


0.47 


-0.33 


Additionally, the table reveals an interesting finding. In 


get more training on the same skills. Tables 3 and 4 


spite of the fact that Information Technology 


together reveal another quite an interesting finding. 


academicians believe that newly graduated Information 


Although they concur that newly graduated Information 


Technology students possess higher competency than 


Technology students have reasonable training on web 


required in skills such as collaboration and 


databases, Information Technology managers believe 


understanding organizational constraints, Information 


that the students should get more training on the same to 


Technology managers believe that the students should 


be more useful for the work of the organization. 
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CONCLUSIONS 

According to Rada (1999), many Information 
Technology jobs go unfilled due to a lack of adequately 
skilled people to fill them. The goal of this paper was to 
address this concern by exploring the skills needed by 
developers of e-Commerce systems. The study proposed 
a framework pertaining to three categories of e- 
Commerce development knowledge — technical, human, 
and organizational — and found some 16 possible high- 
level e-Commerce skills tapping the three knowledge 
areas. The preliminary results reveal that there are some 
agreements and disagreements among Information 
Technology managers and academicians with regard to 
usefulness/importance of the different skills needed for 
developers of e-Commerce systems. The results also 
reveal that there are some similarities and differences 
among Information Technology managers and 
academicians perceptions with regard to perceived e- 
Commerce skills and knowledge of recently graduated 
Information Technology professionals. Furthermore, 
most technical e-Commerce skills comeat the top of the 
mismatch between the usefulness-competency gap as 
perceived by Information Technology managers and the 
same gap as perceived by Information Technology 
academicians. 

Given the rapid change in nature of e-Commerce 
development and consequently the skills needed by 
developers of e-Commerce systems, managers and 
academicians worldwide will value the implications of 
the present empirical investigation. This change is 
projected to have significant impacts on various 
elements of the e-Commerce training/preparation system 
(Heckman 1998). One of the critical challenges facing 
managers and academicians in this regard, hence 
warrant further research, is to understand low-level (as 
opposed to high-level) skills neededby developersof e- 
Commerce systems. 

The practical implications of the present study could be 
many. However, we limit ourselves to two suggestions 
that we offer to academic institutions offering programs 
in Information Technology at this point. The first 
recommendation is that the Information Technology 
curriculum should give more emphasis on technical web 
courses to ensure that graduates have a strong web 
security, web programming, web networking, web 
management, and web site design foundation in order to 
meet the current and future demands of the e-Commerce 
environment. The greatest transition in job requirements 



in the Information Technology market has usually 
occurred for technical knowledge and skills (Todd et al. 
1995). The second suggestion is that Information 
Technology graduates should be equipped with the 
ability to collaborate with work partners and to cope 
with the changing e-Commerce environment and 
demands. Students must be made to realize that the 
academic education is only to prepare them with a 
foundation in Information Technology and that the best 
training it can provide for them is the ability to 
cooperate with others and to learn in whatever area 
necessary. 
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ABSTRACT 

Not-for-profit organizations employ ll%ofall U.S, workers; these organizations are often the recipients of hand-me- 
down hardware and software. This study investigates accounting students' expectations of the information 
technology available to and used by not-for-profit organizations. In this descriptive study, based on two different 
surveys, students had much higher expectations of number of paid staff and amount of hardware than the reality of 
Louisiana nonprofit organizations. Clear discrepancies existed between student expectations of number of software 
applications in use and actual use reported by nonprofits. Accounting students ranked accounting software as the 
most important software for nonprofits, yet only 44yo of nonprofits reported the use of this type of package. Students 
clearly think that not-for-profits in Louisiana are more technology-rich and technology-savvy than the not-forprofits 
report about themselves. 



INTRODUCTION 

This project investigates the expectations of upper-level 
accounting students regarding the availability of 
information technology (IT) in private nonprofit 
organizations in Louisiana. 

Researchers, practitioners, and academicians have been 
calling for changes in the accounting curriculum for the 
past 15 years (American Accounting Association, 1986; 
Accounting Education Change Commission, 1996; 
Russell, et al, 2000). Comments from one university 
that formed a Curriculum Review Team (CRT) and 
surveyed all stakeholders of the accounting department 
included “concern among faculty about the (in)ability of 



the department’s curriculum to prepare students 
adequately for the rapidly changing and complex 
accounting environment they would enter upon 
graduation.” The CRT’s research indicated that “both 
the stakeholders and commentators in the literature 
expressed the view that accountancy teaching is too 
divorced from ‘real-world’ situations . . (Porter, 
1999). 

Yet in the process of upgrading the curriculum to 
include more technology, to require accounting 
information system courses as well as a varying number 
of computer information systems or computer science 
courses, have we led the students to expect more 
technology than is actually available in the “real world”? 
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According to John Palmer Smith (2000), “there are now 
more than 1.5 million nonprofit organizations in the 
United States.” Approximately seven percent of all paid 
employees in the United States work for noiprofit 
organizations. If the full-time equivalent of volunteer 
workers for nonprofits is included, the total accounts for 
about eleven percent of United States workers. Why 
should the researchers study a segment that’s a 
relatively small part of the economy? Today’s 
accountants, indeed all workers, are less likely to hold 
one job over their careers than in past decades. Job and 
career changes increase the likelihood of working in a 
not-for-profitorganization over a worker’s career. Also, 
accountants are often in ccxitact with not-for-profit 
organizations away from their jobs — they are volunteers 
for churches or community organizations, for example. 

Kleen and Foust (2001) found that Louisiana noiprofit 
organizations often had no organization-owned 
computers. Those nonprofits having computer hardware 
frequently used only a very few application software 
packages. Only 44 percent reported using any type of 
accounting software package such as Peachtree, Quick 
Books, etc. 

Companies and organizations in the “for-profit” world 
regularly upgrade their computer hardware and 
software, resulting in personal computers having a 
useful business life of three or four years. Many firms 
give their outdated computers to nonprofits, schools, or 
employees (“New Uses for Old Computers,” 2000). 
Microsoft has instituted a nationwide program to grant 
software to higher education, environmental, human 
service, and civic-related organizations that have 
nonprofit, tax-exempt status, and adequate hardware to 
operate the software (Briggs-Harty, 2000; Lundine, 
2000). However, if hardware is older “hand-me- 
downs,” many nonprofits may not be able to accept 
these software grants. 

To provide software to nonprofit organizations, service 
companies have been developed recently. Software 
companies have been formed that provide online 
fundraising technology, as well as relationship 
(members, donors) management technology (“Program 
helps nonprofits,” 2000). Nonprofits are using the 
Internet for fund raising — not just accq^ting on-line 
donations and gifts, but also conducting online auctions. 
More and more nonprofits are using application service 
providers (ASPs) to save them money and remove the 
need for software maintenance. The ASPs provide the 
software available online and are responsible for 



keeping it updated. The nonprofit’s rental is based on 
number of users or some other use measure (“Nonprofits 
Discover,” 2001). 

An investigation of the literature found no studies 
directly focused on student expectations of information 
technology use in nonprofit organizations. 

PURPOSE 

Not all accounting graduates work for traditional 
corporations; some will have working relationships with 
not-for-profitorganizations. WhetherthelT usedwithin 
not for profits is state of the art, backward, or in between 
is a relevant subject of study. More modem not-for- 
profit organizations may be better able to compete than 
less modem organizations. Studentperception ofthat IT 
level in not-for-profits is also relevant. If student 
perceptions are inaccurate, their training time and job 
adjustments are often longer. 

What are accounting students’ expectations of 
information technology (IT) available to and used by 
nonprofit organizations for which they may volunteer, 
be paid staff, audit accounting records, oc have some 
other type of working relationship? The focus of this 
research is to determine what expectations accounting 
students in Louisiana public universities have regarding 
the information technology available to and used by 
Louisiana nonprofit organizations. How well do these 
expectations match the reality of IT usage in Louisiana 
nonprofit organizations? The findings of a survey of 
accounting students, conducted by the researchers, were 
compared to results of a recent study of actual Louisiana 
nonprofit organizations.” 

METHODOLOGY 

The researchers constructed a survey instrument to 
determine accounting students’ expectations of 
information technology availability and use by nonprofit 
organizations. The instrument paralleled an instrument 
used in an earlier study that focused on determining how 
nonprofits in Louisiana reported using information 
technology to support their activities and mission. The 
current instrument was designed for accounting students 
as the responding group. Students in selected junior and 
senior-level accounting classes at two Louisiana public 
universities participated. Classification variables for the 
student respondents were gender and age group. The 
researchers obtained approval for the current study 
through their university’s Human Subjects Institutional 
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Review Board procedures. This data collection does not 
represent a random sample, but is rather a pilot study. 
Only limited attempts will be made to conduct 
hypothesis tests within student respondents, and none 
between student expectations and actual not-for-profit 
responses. Where hypothesis test results do appear in 
the analysis below, it is for two purposes: guiding 
survey redesign and validating researchers’ a priori 
expectations. 

Survey questions focused on the following expectations 
of information technology usage among nonprofit 
organizations: 

• Hardware and software availability 

• Productivity software packages in use 

• Monitoring of finances/program results (accounting 
systems) 

Analysis of the data was completed using SPSS 
software. In addition to descriptive statistics that 
illustrate which areas of IT support are most commonly 
expected, cross tabulation and correlation tools were 
used to show how (or whether) the extent of IT 
expectations varies with the organizations ’ classification 
data. Linear regression relates organization paid staff 
size to number of software applications used. The 
students’ expectations were compared against the actual 
findings from the previous study of nonprofit 
organizations in Louisiana. 

Investigative questions to be dealt with in the statistical 
analysis include: 

• Student expectations for size of not-for-profit 
organizations’ paid staff and volunteers 

• Student expectations of computer availability and 
operating system currency 

• Student expectations of number of software 
applications used by not-for-profits 



• Student expectations regarding use of accounting 
software 

• Student expectations of importance of software 
applications used 

• Student expectations about other information 
technologies used by not-for-profits 

FINDINGS 

Fifty-seven accounting students in three accounting 
classes responded to the survey. Of these students, 
77.2% were female. Four-fifths (80.7%) were under the 
age of 25. 

Student Expectations of Not-for-Profit 
Organizational Staff 

As shown in Table 1, although over 90% of students 
expected a minimum of three or more paid staff in a 
nonprofit organization, 58.3% of the nonprofit 
organizations surveyed reported no paid staff. Students 
also had higher expectations of number of volunteers 
active in the organization than reported in the survey of 
actual nonprofit organizations. As shown in Table 2, 
contrary to student expectations, more than 50% of 
nonprofits reported 10 or fewer active volunteers. 

Student Expectations of Computer Availability and 
Operating System Currency 

Although 98% of students expected the nonprofit 
organization to own at least one computer, over 50% of 
nonprofits surveyed reported no organization-owned 
computers. Over 70% of students expected at least three 
organization-owned computers; only 24.6% of 
organizations reported a minimum of three computers. 
(See Table 3.) 

Over one-third (35.1%) of students surveyed expected 
the computers to be exclusively PCs. Another 61.4% 
expected mostly PCs. Only 1.8% expected mostly 
Macs. No students expected exclusive use of Macs by 
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TABLE 1 

NUMBER OF PAID STAFF 



Paid Staff 


Student Expectations Percentage 
(n = 57) 


Actual Nonprofit Findings Percentage 
(n = 163) 


0 


7.0 


58.3 


1-2 


1.8 


12.3 


3-5 


45.6 


11.7 


6 or more 


45.6 


17.2 


No response 


0.0 


.6 


Totals 


100.0 


100.0 





TABLE 2 

NUMBER OF ACTIVE VOLUNTEERS 




Number of Volunteers 


Student Expectations Percentage 
(n = 57) 


Actual Nonprofit Findings Percentage 
(n = 163) 


10 or fewer 


21.1 




50.9 


11-20 


29.8 




11.7 


21-30 


29.8 




9.2 


Over 30 


19.3 




27.0 


No response 


0.0 




1.2 


Totals 


100.0 




100.0 



TABLE 3 

COMPUTERS OWNED BY NONPROFIT ORGANIZATIONS 


Number of Computers 


Student Expectations Percentage 
(n = 57) 


Actual Nonprofit Findings Percentage 
(n = 163) 


0 




1.8 


52.1 


1-2 




26.3 


23.3 


3-5 




42.1 


12.9 


6 or more 




29.8 


11.7 


No response 




0.0 


0.0 


Totals 




100.0 


100.0 



nonprofits. What operating system do students expect 
the nonprofits to be using on their computers? The 
results shown in Table 4 illustrate that students 
predominantly expected the more recent versions of 
Microsoft Windows to be in use. For the smaller 
number of students expecting Macintosh operating 
systems (not illustrated in Table 4), the respcxidents 
once again indicated they expected the more recent 
versions of the operating systems to be in use. (There 
were no comparable questions in the earlier survey of 
not-for-profits.) 



Do students expect laptop computers to be in use in 
nonprofit organizations? As shown in Table 5, almost 
two-thirds of respondents expected mostly desktop 
systems to be owned by the organizations; only 3.5% 
expected mostly laptop systems. Over two-thirds 
(7 1 .9%) of students expected the computers to be in use 
in the organization’s office. No students expected the 
organization’s computers to be in use in organization 
workers’ homes. 
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TABLE 4 

STUDENT EXPECTATIONS OF OPERATING 
SYSTEMS IN USE ON PCS 
(MACS EXCLUDED) 

(n = 55) 



Operating System 


Percentage* 


Windows NT, XP, ME, or 2000 57.9 


Windows 98 


61.4 


Windows 95 


24.6 


MS-DOS 


12.3 


Other 


7.0 


^Totals are more than 
answers selected. 


100% because of multiple 



TABLE 5 

STUDENT EXPECTATIONS OF USE OF 
LAPTOP AND DESKTOP COMPUTERS 
(N = 57) 


Type of System 


Percentage 


Exclusively desktops 


29.8 


Mostly desktops 


64.9 


Mostly laptops 


3.5 


Exclusively laptops 


0.0 


No response 


1.8 


Totals 


100.0 



Students do expect nonprofit organization members/ 
volunteers to use their home computers for organization 
work, as shown in Table 6. Only one of the 54 respon- 
dents answering this question expected mostly Macs. 
All others expected either exclusively PCs or mostly 
PCs. Almost two thirds of respondents (64.9%) expected 
mostly desktop systems in use by volunteers. 

TABLE 6 

STUDENT EXPECTATIONS OF 
INDIVIDUALLY OWNED COMPUTERS 
USED IN THE ORGANIZATION 
(N = 57) 



Number of Computers 


Percentage 


0 


8.8 


1-2 


26.3 


3-5 


29.8 


6-8 


14.0 


More than 8 


21.1 


Totals 


100.0 



Student Expectations of Software Usage 

What software applications do students expect the 
nonprofit organizations to be using? As illustrated in 
Table 7, student expectations of software used in 
nonprofit organizations was much higher than what is 
happening in reality. Only word processing, e-mail, and 
spreadsheet software were reported in use in more than 
50% of nonprofits, although students expected those 
percentages to be in the 80 to 90 percent or more range. 
Figures la and lb further illustrate the discrepancy 
between student expectations of number of software 
packages in use and the reality of nonprofit organization 
software usage. The mean number of applications 
expected by students was 5.11; the mean reported by 
actual not-for-profits was only 3.33. 

What components of accounting packages do students 
expect nonprofit organizations to use? As illustrated in 
Table 8, 53 of 57 respondents (93%) expected the 
nonprofit organizations to use an accounting package 
and 91% the use of more than one component of the 
package. This again reflects much higher expectations 
of accounting package usage than reported in actual 
nonprofit organizations. The reality is that less than half 
use an accounting software package, and only 35% use 
more than one element of it. Figures 2a and 2b clearly 
illustrate this discrepancy between expectations and 
reality. The distribution of number of accounting 
package elements used are dramatically different. 

Based on a cross-tabulation of “Number of individually 
owned computers used” versus “Use of an accounting 
package,” students expected a positive relationship. The 
percentage of not-for-profits using accounting packages 
rises with the number of computers, exceeding 90% for 
the most number of computers category. The data 
revealed no relation between type of computer (PC 
versus Mac) or type of computer system (desktop versus 
laptop) and the likelihood of using an accounting 
package. 

How do students rank the importance of the software 
applications used within nonprofit organizations? Table 
9 below illustrates that while accounting students ranked 
accounting software first in importance, nonprofit 
organization respondents ranked it fourth in importance. 
While nonprofits ranked e-mail third in importance, 
higher than spreadsheet and accounting packages, 
students ranked it sixth in importance. 
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TABLE 7 

SOFTWARE APPLICATIONS USED WITHIN NONPROFIT ORGANIZATIONS 



Software Package 


Student Expectations Percentage 
(n = 57) 


Actual Nonprofit Findings Percentage 
(n = 163) 


Database management 


53.0 


45.0 


Spreadsheet 


82.0 


51.0 


Word processing 


88.0 


74.0 


Contact management 


35.0 


Not asked 


Newsletter/desktop publishing 


68.0 


42.0 


Accounting package 


82.0 


44.0 


E-mail 


93.0 


56.0 


Web-site building package 


44.0 


17.0 


Tax preparation 


40.0 


Not asked 



FIGURE lA 

STUDENT EXPECTATIONS OF SOFTWARE 
APPLICATIONS IN USE IN NONPROFIT 
ORGANIZATIONS 

16 
14 
12 
10 
8 
6 



FIGURE IB 

ACTUAL SOFTWARE APPUCATIONS IN USE 
AS REPORTED BY NONPROFIT 
ORGANIZATIONS 





Software Applications Used Software Applications Used, Actual 



TABLE 8 

USE OF ACCOUNTING SOFTWARE 



Package 


Student Expectations Percentage 
(n = 57) 


Actual Nonprofit Findings Percentage 
(n = 163) 


General ledger 


86.0 


40.5 


Accounts payable 


77.0 


33.7 


Check writing 


84.0 


Not asked 


Payroll 


81.0 


22.1 


Pledges/Memberdiip receivable 


75.0 


22.1 


More than 1 component 


91.0 


35.0 


No accounting package 


5.0 


36.2 


No response 


2.0 


17.8 
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FIGURE 2A 

STUDENT EXPECTATIONS OF ACCOUNTING 
SOFTWARE APPLICATIONS IN USE IN 
NONPROFIT ORGANIZATIONS 



FIGURE 2B 

ACTUAL ACCOUNTING SOFTWARE 
APPLICATIONS IN USE AS REPORTED BY 
NONPROFIT ORGANIZATIONS 




Accounting Package Elements Used Accounting Package Elements Used, Actual 



TABLE 9 

RANKING OF IMPORTANCE OF SOFTWARE 



Software Package 


Student Expectations Ranking 


Actual Nonprofit Findings Ranking 


Database management 


3 


2 


Spreadsheet 


4 


5 


Word processing 


2 


1 


Contact management 


8 


Not asked 


Newsletter/desktop publishing 


5 


6 


Accounting package 


1 


4 


E-mail 


6 


3 


Web-site building package 


7 


7 


Tax preparation 


9 


Not asked 



Student Expectations of Other Technology Usage 

Students were also asked about their expectations 
concerning the use of such items as pagers, cell phones, 
fax machines, and personal digital assistants (PDAs) in 
nonprofit organizations. As illustrated in Table 10 
below, almost 90% of students expected the use of fax 
machines, but a minimal 3.5% expected use of PDAs. 
(There were no comparable questions in the survey of 
not-for-profits.) 



TABLE 10 

STUDENT EXPECTATIONS OF USE OF PAGERS, 
CELL PHONES, FAX MACHINES, AND PDAS 
(n = 57) 



Device 


Percentage 


Pagers 


40.4 


Cell Phones 


61.4 


Fax Machines 


89.5 


Personal Digital Assistants 


3.5 
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Correlations 

When the student e^qjectations survey results alone were 
analyzed, there were no significant correlations between 
expected size of organization and the rank of the 
software in use. As expected, the correlation between 
one application's rank and another application's rank 
were universally negative. Number of volunteers was 
positively correlated (r= .282) with use of pagers. Use 
of cell phones was positively correlated (r= .285) with 
use of pagers. 

Number of paid staff was positively correlated with use 
of database management (r = .304), spreadsheet 
software (r = .282), and accounting software (r = .282). 
Not surprisingly, the use of one part of an accounting 
package is correlated with use of the other parts. 

Regressions 

The researchers conducted a linear regression with paid 
staff size category as the independent variable and 
number of software applications used as the dependent 
variable (even though both variables are ordinal and the 
sample is not random). Student data revealed an 
apparent relationship between the number of 
applications used by a noiprofit organization and its 
number of paid staff. Findings revealed students 
expected organizations with more paidstaff to use more 
applications. With each change in staff size, 
applications rise by almost one. This positive 
relationship matches researcher expectations. 

CONCLUSIONS AND RECOMMENDATIONS 

Study findings revealed that accounting students' 
expectations of the information technology available to 
and used by not-for-profit organizations do not match 
reality. Students clearly expected more hardware 
available than what was reported by Louisiana noiprofit 
organizations. Clear discrepancies existed between 
student expectations of number of software applications 
in use and actual use reported by nonprofit 
organizations. Likewise, students had much higher 
expectations of accounting package usage than reported 
in actual nonprofit organizations. While accounting 
students ranked an accounting package as the most 
important software application for nonprofit 
organizations, only 44 percent of nonprofits reported 
using any type of accounting software; the noiprofit 
organizations ranked accounting software fourth in 
importance, behind word processing, database 



management, and e-mail. The student data revealed an 
apparent relationship between the number of 
applications used by a noiprofit organization and its 
number of paid staff; students expected organizations 
with more paid staff to use more applications. 

The researchers plan further research related to not-for- 
profit organizations and student expectations The not- 
for-profit survey instrument needs revision to add 
hardware questions as were used in the student survey. 
Future research should include random samples of 
accounting students in parallel with surveys of not-for- 
profit organizations. Perhaps a separate survey for 
CPAs could be conducted as the reality chedc. 

IMPLICATIONS FOR EDUCATORS 

Louisiana accounting students clearly think that not-for- 
profits in Louisiana are more technology-rich and 
technology-savvy than the not-for-profits report about 
themselves. The immediate lesson from this is to 
prepare students for this low-tech reality. This may take 
only a segment of a lecture, or a short exercise. 
Students in accounting systems classes can have an 
opportunity to experience the low-tech reality through 
real-world not-for-profit organization projects. 
Internship opportunities in not-for-profit organizations 
may also provide this reality check the classroom alone 
does not provide. Service learning may provide students 
with a much more realistic view of what technology is 
available to and utilized by nonprofit organizations. The 
findings of this study also have implications for various 
other disciplines. 
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A QUANTITATIVE REVIEW OF 
PROGRAM ATTRIBUTES AND OFFERINGS 
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ABSTRACT 

This paper reviews several previous studies of course offerings in a large number of Electronic Business/Commerce 
concentrations in both MBA and MS programs. Results from these earlier studies indicate that there is no apparent 
consensus in what knowledge is core to the Electronic Business/Commerce concentration. This study takes data 
from these previous studies and looks at the number and types of courses offered in Electronic Business/Commerce 
concentrations and compares and contrasts what is offered in MS and MBA programs. This analysis reveals that 
there are some trends that can be identified in Electronic Business/Commerce concentration offerings. Certain core 
knowledge is offered in many of the programs reviewed, but the level presented varies greatly. The analysis also 
reveals that there is some correlation among the most common course offerings. A key discriminant is the presence 
of an E-business marketing course. When this course is present, it often signals the co-occurrence of other specific 
additional courses, forming a package of core course content. When the course is not part of the program, there is 
significant variation among the remaining course offerings of the program. 



INTRODUCTION 

This paper discusses some of the issues and concerns 
voiced by those tasked with planning and developing 
Electronic Business or Electronic Commerce programs 
at schools, and practitioners or students trying to find a 
“good” E-business program. The main issue seems to 
revolve around the question of what should an 
Electronic Commerce/Business program present as key 
elements. This is not a new discussion, but it is one that 
has still not been satisfactorily resolved. This paper 
builds on material developed in three prior studies and 
applies several statistical tools in an attempt to answer 
this question. The task is complicated by the fact that 
Electronic Business/Commerce programs, although 
most often found in MBA or MS in Information 
Systems business programs, can also be found in 
Marketing, Arts and Science, Computer Science, and 
Engineering programs as well 



BACKGROUND 

For the purposes of this paper, academic programs with 
the names Electronic Business, Electronic Commerce, 
E-Business, E-Commerce, Internet Commerce, et cetera 
will be referred to generically as E-business. 
Organizations continue to be focused on the potential of 
E-business to increase their competitiveness and 
improve profitability (Payne, 200 1). Four recent studies 
Novitzki (2002), Whitten and Stephens (2001), Siau 
and Davis, (2000), and Sendell (1999), presented 
results of surveys or analysis of E-Business programs. 
They identified a large and diverse number of course 
offerings across the schools and programs studied. They 
presented either lists of courses or summary tables that 
describe the varied curricular offerings found and 
discuss their impact. There are many points made, but a 
consistent theme is the wide variety and number of 
courses offered in the various programs makes it 



Proceedings of the 17^ Annual Conference of the International Academy for Information Management 



35 



41 



BEST COPY AVAILABLE 



difficult to determine what is key E-Business 
knowledge. 

A study done by Mitchell and Strauss (2001) identified 
several skill and cognitive based clusters that they felt 
were common to many programs. King et al (2001) and 
Etheridge et al (2001) presented findings which 
indicated that while there maybe some broadclusters of 
knowledge, there is still considerable variety in course 
offerings between schools and considerable change in 
offerings from year to year in this rapidly evolving field. 
King (2001) also reported that programs are often driven 
more by faculty skill and stakeholder wants rather than 
academic focus or need. 

There have been several articles (Herrmann and Pemul 
(1999),Teo and Too (2000), and Williams et al (2000)) 
which indicate that while there is significant difficulty 
in identifying the key issues in dealing with E-business, 
there is an even greater problem developing the 
programs that can best communicate this information to 
students and practitioners in the field. 

CURRENT STUDY 

There is, as was described above, a wide and varied 
perspective in E-business program offerings. Two 
characteristics that have not been studied quantifiably is, 
are there real clusters of knowledge as Mitchell and 
Struss (2001) contend, and is there a consistency in what 
is offered across programs? This study looks at the 
results of three previously mentioned studies (Novitzki 
(2002), Whitten and Stephens (2001), and Sendell 
( 1 999)). It combines the schools and programs used into 
a large sample of programs to determine such critical 
points as what does an average E-business program look 
like? What is the probability that a program will have 
one of the most common courses? What is the 
probability of a program having all of the most common 
courses? Is there any correlation between course 
offerings in programs that indicate a consistency of 
offerings or theory? Witii the wide variation in both 
school focus and degree offered, a key question that 
must also be considered is, how much do tiiese factors 
impact the curriculum that is presented to students? 

Sendell (1999) and Whitten and Stephens (2001) 
reported on E-business programs in Association to 
Advance Collegiate Schools of Business (AACSB) 
International accredited schools which are a subset of all 
schools offering these programs, and provided no 
comparison to other schools. Novitzki (2002) looked at 



a wider range of schools including AACSB schools and 
degree programs certified by Certified E-Commerce 
Consultants, many of which were not AACSB 
accredited. 

METHODOLOGY 

This study combines the schools and programs reviewed 
in the previous studies to develq) a statistically 
significant sample of E-business programs. The data 
was reviewed for all the studies and duplicate programs 
were eliminated from the study. This reduced the 
number of schools being reviewed to 163. Offerings at 
these schools were broken down into MBA or MS 
program as appropriate due to the differences inherent 
in the focus, course offerings, and course requirements 
in these two programs. 

FINDINGS 

The most widely used courses for MBA and MS 
programs were tabulated and the results placed in Table 
1 . The table contains the top ten courses offered in the 
two degrees from a list of over 100 unique courses. The 
table also shows the difference in E-business 
concentrations between MBA and MS programs. Only 
three of the four top MBA courses are in the top four for 
MS programs, and the differences in offerings between 
other courses listed shows how degree focus affects 
course offering^. The table also clearly shows the wide 
variation in program offerings within each degree 
program. In the MBA programs only one course, E- 
business Marketing, is being offered in more than 50% 
of the programs. The MS situation is similar showing 
only two courses. E-business Marketing and E-business 
Technology being offered in more than 50% of the 
programs studied. 

No course was offered in 1 00% of the schools reviewed. 
If we look at the top four courses in the MBA programs 
(the most common number in an MBA concentration), 
the highest presence in programs for a course was 78% 
and the lowest was 34%. Given the breadth and number 
of offerings available noted above, there is a significant 
variation in what is being offered in programs and some 
schools could have an entirely different set of courses 
than the ones shown here. The fact that no specific 
course was required in all MBA programs also means 
that the focus of the concentration can change 
significantly depending on the individual school and 
program. 
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TABLE 1 

MOST COMMONLY OFFERED COURSES 



DISCUSSION 



Most Common Courses 


MBA 


MS 


E-Business Marketing 


78% 


76% 


E-Business Strategy 


49% 


12% 


E-Business Technology 


34% 


73% 


Legal Issues and Ethics 


31% 


46% 


Introduction to E-Business 


38% 


29% 


Supply Chain Management 


23% 


4% 


E-Business Programming 


4% 


13% 


E-Business Security 


12% 


13% 


E-Business System Analysis 


6% 


9% 


and Design 






Datami ni ng/DBM S 


9% 


8% 



The MS groupings also show wide variation in 
offerings, and Table 1 summarizes those as well. If we 
look at the top six courses in these programs (the most 
common number in an MS concentration), the top two 
are found in over 70% of the programs, but the lowest 
was found in only 13% of the programs. This wide 
range was due to several reasons. The schools offering 
an MS in E-business were varied with many not being in 
business schools. Due to differences in the programs’ 
focus, there were more unique courses offered. Thus an 
E-business concentration in a Computer Science 
program had proportionally more technical and less 
business courses than a program offered in a business 
school. Similarly, a Masters in E-commerce Manage- 
ment in a School of arts and Science had a more 
balanced list of course offerings. An MS in E-business 
Marketing would have another mix of classes, while an 
E-business concentration in an MS in Information 
Technology from a school of Applied Science had yet 
another mix of course offerings. 



One of the most serious problems identified is the 
complete lack of consistency in what is offered between 
programs whether they are an MBA or MS. If you take 
the numbers presented in Table 1 for the various 
programs, then the statistical likelihoodof two programs 
offering the four most common courses to their students 
is barely 5% for MBA programs and less than 1% for 
the six courses in the MS programs. For MS programs, 
however, the situation is much better when you consider 
a subset of the courses. If only the top three courses are 
considered, more than 25% of the programs offer the 
same courses. This comparison does not help the MBA 
programs as much with only 12% of the MBA programs 
containing the top three courses. 

Table 2 shows that E-business MBA concentrations, as 
currently offered by schools, have little consistency, and 
as a result it is extremely difficult to identify what the 
core knowledge of the E-business concentration is. 
Employers can have little hope expecting consistent 
knowledge when hiring from graduates of more than one 
E-business program. Students wanting an E-business 
concentration besides deciding whether they want an 
MBA or MS, must clearly research each program to see 
if it provides the skill set that they are looking for. 

This study also reveals several points about E-business 
programs that have both good and bad indications. 
There is a set of courses which are offered at some 
programs, but the number offering them is currently 
very small. In the MS programs the presence of the 
same three courses in more than 25% of the programs 
indicates the development of a core. Overall, however, 
the number of offered courses is large and there is little 
consistency. With more than 100 unique courses being 



TABLE 2 

CUMULATIVE PROBABILITIES OF PROGRAM COURSE OFFERINGS 



MBA MS 

Condition Programs Programs 

Probability of a school offering the most commonly offered course 0.78 0.76 

Probability of a school offer the two most commonly offered courses 0.38 0.55 

Probability of a school offering the three most commonly offered courses 0.14 0.25 

Probability of a school offering the four most commonly offered courses 0.05 0.07 

Probability of a school offering the five most commonly offered courses (MS only) 0.01 

Probability of a school offering the six most commonly offered courses (MS only) <0.01 
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offered in these programs, it is unlikely that a standard 
core can develop unless this number is significantly 
reduced. Even though the schools sampled diffoed in 
accreditation and base degree, many of the same courses 
appear in both degrees. Almost 70% of the course 
offerings are common to both the MBA and MS 
degrees. There are also indications that the grouping of 
courses is becoming more consistent. If subsets of 
schools are considered, then the probability of receiving 
a standard set of four courses increases considerably. 
For example, if large AACSB schools and small 
AACSB schools are considered separately, then the 
probability of receiving the same four courses in their 
MBA programs increases to 1 6% and 1 2% respectively. 
For the MS programs there is improvement as well, but 
it is not nearly as much. When the large AACSB 
programs are separated from the small AACSB 
programs, the probability of receiving the same six 
courses increases to 3% and 1% respectively. 

There are some differences in offerings between 
AACSB and non- AACSB schools. The top four courses 
in non- AACSB program differ from the small AACSB 
programs, but tend to match those of the large AACSB 
schools. One reason postulated for this is, that at 
schools with small MBA programs. E-business is often 
taught by faculty from other base disciplines. As a 
result offerings often tend to build on the base discipline 
rather than having a strategic E-business focus. 

The most interesting issue, that requires more study, is 
why the E-commerce Marketing course and not the 
Introduction to E-business course seems to be a major 
disciminant, as well as the most common course offered 
in both degrees. Nothing in the literature postulates why 
that should be true. Yet when the E-commerce 
marketing course is present, MBA programs are more 
likely to have one or more the remaining three courses 
than if the course is not present. Similarly the presence 
of the E-commerce marketing course in an MS program 
increases the likelihood of other courses, but not to the 
extent seen in the MBA programs. 

CONCLUSION 

E-business programs have been developed in several 
disciplines. Material presented in E-business 
concentrations often seems more linked to the base 
discipline than to the issues and concerns of E-business. 
There is wide variation in what different schools present 
as E-business programs, but initial statistical analysis 
indicates that there is in fact a grouping of knowledge 



and courses appearing in programs that indicates an 
internal consistency that is not apparent when looking at 
the raw data. 

The field is still developing and the changes in technical 
capabilities impact E-business programs as well as 
organizations. There is obviously still no general 
consensus on what should be the core knowledge in the 
concentration. Will these programs eventually develop 
a standard core? They probably will in the future, but 
faculty are the ones who will have to make it happen. 
We must design programs that are truly responsive to 
the needs of student, but which are also academically 
sound and which provide the knowledge that our 
graduates will need in the future as well as now. Many 
current programs seem to have been created more in 
response to specific employer needs or faculty ddlls 
rather than as a result of a detailed analysis of critical 
elements and knowledge in the field. If we fail to move 
beyond the present confused state of programs, it is 
unlikely that the field will develop as a major 
independent field of study. 
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ABSTRACT 

This panel will explore challenges and opportunities which MIS faculty encounter in the production of scholarfy 
research and the development and dissemination of curricula. Discussion will address two issues including: (1) the 
trade-off between research and teaching in the promotion and tenurereview process; and, (2) the effects of resource 
constraints on research activity and teaching endeavors. 



Institutional demands exist which compel educators to 
focus their energies toward teaching or research. 
Universities are typically classified as research or 
teaching institutions. Research universities place greater 
emphasis on research than on teaching during promotion 
and tenure reviews. Teaching is considered important at 
research universities. Faculty must achieve above 
satisfactory teaching reviews in order to be considered 



for upward mobility. Faculty atresearch universities are 
often given reduced teaching loads to compensate for 
the added research demands. 

At research institutions, scholarly research is typically 
assessed using ranking systems where journals and 
academic conferences are evaluated based on prestige 
and acceptance rates. The ranking of journals is 
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accomplished on a four-point scale — A, B, C, and D. 
This classification schema forms the basis for 
assessment in the research-based institution. Academics 
are evaluated based on the quality and quantity of 
research activity. Institutions often maintain specified 
requirements such as 4 As or 6 Bs necessary for 
advancement to the subsequent academic rank. 

Teaching universities enphasize curricular issues, 
teaching evaluations, and innovation in the classroom 
during the promotion and tenure process. Teaching is of 
greater consequence than research. Maintaining an 
active research agenda is important but the degree of 
activity varies across institutions. In addition, even if 
continuous involvement in research that translates into 
a consistent stream of peer reviewed journal articles is 
also a requirement for promotion at these universities, 
publication in “A” journals is less important. The 
faculty members at teaching universities typically have 
a higher teaching load than faculty at research 
institutions. Teaching load reductions for significant 
research activity are awarded less frequently at teaching 
institutions than at research institutions and when 
awarded, rarely match those typically observed at 
research institutions. Teaching innovation and novelty 
in the classroom is rewarded. The use of technology to 
facilitate learning is often deemed significant. 

Academic institutions, both public and private, are 
impacted by various resource constraints including 
financial, staffing, technological, and time. Budget 
cutbacks exist at state institutions. Private institutions 
are holding spending fiat. Many institutions are looking 
toward its faculty to generate revenue through grant 



writing. Administrations seek to generate revenue 
through increased class size. Technological constraints 
also exist. In a field of study, where change is constant 
and technologies frequently modify, both internal and 
external pressures may challenge MIS educators to offer 
instruction on the utilization and manipulation of the 
latest technologies. The use of new technologies 
requires financial resources to purchase the latest 
hardware and software. In addition, faculty time and 
intellectual resources are required to “ramp up” on these 
new technologies. Faculty must not only learn the 
technology but also transfer this knowledge to 
undergraduate and graduate students. 

Maintaining exemplar educational information systems 
programs necessitates highly educated faculty, 
challenging curricula, new technology, and state-of-the- 
art facilities. Each component requires resources to 
remain competitive. Trade-offs exist between: (1) 
increasing faculty salaries, (2) compressing faculty 
salaries, (3) employing new faculty, (4) paying for 
faculty conference participation, (5) establishing new 
academic programs, (6) hiring additional graduate 
teaching assistants, (7) introducing new technology, and 
(8) building new facilities. These factors impact faculty 
at both research and teaching institutions, often making 
it more difficult to perform satisfactorily in the areas 
that carry the most weight in promotion and tenure 
processes: research and teaching. 

A discussion pertaining to these issues will be 
undertaken. Intellectual exchange stimulated through an 
environment of open discussion is proposed for this 
panel presentation. 
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ABSTRACT 

Team-based student projects have become an intrinsic part of coursework in information technology courses. The 
rationale is that once students enter the work environment they will be required to work in teams. Challenges of 
teamwork are reviewed and factors influencing student team performance are identified. An approach is described 
which deals with four categories of concern when incorporating a team-based project in information technology 
courses, namely planning, execution, evaluation and team process improvement. The approach stresses the 
importance of utilizing best practices, as available in international standards, for team projects of a full term 
duration that is based on a case study. It is argued that such an approach contributes to bridging the gap between 
student project teamwork based on limited case studies and real-world projects conducted in the competitive business 
environment of today. The best practices and standards used in 34 classes for eight courses of a master's program 
in computer and information systems are summarized in the paper. A total of 150 teams participated in these courses. 



INTRODUCTION 

Team-based student projects have become an intrinsic 
part of coursework in information technology (IT) 
courses (Meyer, 2001; Stephens, 200 1 ). The rationale is 
that once students enter the work environment they will 
be required to wcrk in teams. Working in a team context 
challenges team members in a number of ways, such as: 

• Teams are composed of individuals with different 
technical skills, cultural backgrounds, behavioral 
characteristics, cognitive styles and learning abilities. 

• Performance of team members is influenced by the 
level of teamwork and in- field experience, knowledge 
of the application domain, pressures of schedule, 
geographical dispersion, full-time or part-time study. 

Various authors have reported on project teamwork in 
the academic environment (Mennecke et al,1998; 
Jovanovic et al, 1998; McKendall, M., 2000; Stephens, 
C. and O’Hara, M., 1999). However, Stephens (2001) 
has analyzed the literature on student teams finding that 



a limited number addressed the issue of effective 
teamwork in the academic context. The fact is that most 
research equates the e?q3erience of student teams with 
that of professional workgroups despite the fact that the 
characteristics of the two types of teams are very 
different. For instance, the control structure, 
mechanisms for monitoring and managing, and the 
composition of the two types of teams are significantly 
different. 

A team approach to student projects over a full term has 
formed part of most of the courses in the master’s 
program in computer and information system at the 
University of Detroit Mercy. Our experience in 34 IT 
courses, where teamwork formed part of the pedagogy, 
has confirmed experiences reported by others (Jones, 
1996; Stephens, 2001) regarding the importance of 
factors such as: 

• Thorough planning of the team project 

• Appropriate interventions to improve effectiveness of 
student teams 
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• Efficient and effective project team leadersh^ 

• Active team member commitment and participation 

• Appropriate groupware technologies. 

In addition it is our experience that using best practices 
as available in international standards enhances the 
quality of the project, the deliverables, and provides a 
mechanism to enhance the value of the student team 
experience once a student enters the marketplace. We 
argue that the approach presented here contributes to 
bridging the gap between student project teamwork 
based on limited case studies and real-world projects 
conducted in the business environment. By adopting a 
team approach in projects we intend to support and 
enhance a student’s ability to learn by absorbing the 
subject matter individually and then applying the 
knowledge gained in a project as part of a team. 

Team process categories structure the aspects relevantto 
planning and implementing a team-based, project- 
oriented process, referred to in this paper as the team 
approach, as part of the course pedagogy. Four team 
process categories are described that concern faculty 
when incorporating a team-based project in information 
technology courses, namely planning (in Section 2), 
execution (in Section 3), evaluation (in Section 4), and 
team process improvement (in Section 5). The processes 
described in the Project Management Body of 
Knowledge (PMBOK) (PMI,2000) have proved very 
useful for detailing the activities of planning and 
executing team-based projects. Team process 
improvement has been done in accordance to practices 
of the Team Software Process (TSP) (McAndrews, 
2000) and processes recommended in the ISO 15504 
(1998). Section 6 summarizes the relevant industry 
standards and best practices used in a range of courses 
with 150 participating teams, and draws some 
conclusions. 

PLANNING CATEGORY 

Activities in this category deal with aspects that should 
be considered when preparing to incorporate the team 
approach in a course. A number of questions must be 
resolved, such as: 

• What do you include under planning? 

• Factors influencing choice of case study 



• Do you have industry involvement? 

• What Standards and best practices are used? 

• Project materials used 

• How do you measure outcomes? 

• What software is used? 

• Do you support classroom teaching with virtual 
teaching and learning aids? 

• Do you maintain a team process database? 

• What are your project deliverables? 

Planning a team-based project within the context of the 
course syllabus requires answers to these questions and 
attention to aspects such as the ones discussed next. 

Faculty should determine the standards and best 
practices applicable for a course with a team project. 
These standards are relevant for both the course specific 
teamwork and the project management tasks to bring 
the teamwork to successful conclusion. Supporting 
materials must be prepared for the team project, 
including tutorial material describing the team project, 
prerequisite knowledge for each team assignment, the 
case study to be used, the team assignments, the formats 
ofproject deliverables (e.g., a requirements specification 
document, or a design document), and related informa- 
tion. 

Where appropriate an industry specialist could be 
identified and invited to participate in the course and 
team project. In some projects industry involvement is 
substantial in that the case study used for the team 
project is provided and a measure of support is provided 
for the teamwork. An example is a case study on the E- 
Business System of a division of an automotive 
company which was used in two courses (Steenkamp et 
al, 2002). In other cases the involvement is in the form 
of presentations on the application domain of the 
project. 

Teamwork related forms, such as a student background 
evaluation (SBE) form, a team composition form, peer 
evaluation form, and related forms must be designed. 
Some generic formats have been developed and are 
available for use in project-based courses. 
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The planning and development of the course website 
needs to be done once some key mato-ials are ready. At 
first support in establishing such a course website was 
provided by the Instructional Design Center at the 
university. More recently the university subscribed to 
the Blackboard e-learning software platform and faculty 
members are now able to develop their own course sites, 
with the necessary training provided by the Instructional 
Design Studio. Other e-leaming software platforms are 
available, such as the Learning Space environment used 
at the ITESM in Montarey, Mexico wiiich is based on 
IBM’s Lotus Notes (Murillo, 1998). 

An important aspect in the planning category concerns 
the logistics arrangements for teamwork (rooms, 
projection equipment and internet-ready computer 
facilities, site visits where q^propriate, and the like). 

Table 1 summarizes typical activities in the planning 
category. 

THE PROJECT EXECUTION CATEGORY 

Once a course starts and a team project is due to 
commence activities in this category need consideration. 
Aspects include initiating the team project in class, 
coordinating teamwork, consulting with teams during 
the project life cycle, and mechanisms for monitoring 
teamwork. For teams to be successful in performing the 
technical tasks of a project it is imperative that they 
adopt a sound development methodology, with rigorous 



coverage of development process, representation 
schemes, notations and format of deliverables relevant 
to the particular course. 

Each of the aspects in the project execution category is 
reviewed next. Table 2 provides a summary of project 
execution activities from the perspective of the faculty 
member. 

Project Initiation 

This aspect includes activities such as establishing the 
teams, and orienting them regarding the case study and 
teamwork. Some questions regarding initiating a team 
project include: 

• How is the project initiated? 

• How are teams compiled? Team size? 

• How are team leaders identified? 

• How are standards and best practices introducedto the 
teams? 

• Do you conduct team tutorials? 

• How do you monitor progress? 

In some cases a prerequisite course may have provided 
adequate coverage of team collaboration and virtual 



TABLE I 

ACTIVITIES IN THE PLANNING CATEGORY 



Planning Category 

1. Select a appropriate case study for the project that integrates the team assignment. 

2. Obtain industry involvement (case study) and arrange fcx* presentation in course session. 

3. Identify appropriate references on teamwork and the virtual team. 

4. Identify appropriate references that orient, guide and support culturally diverse teams. 

5. Identify and obtain appropriate groupware tools to support teamwork. 

6. Identify and obtain appropriate software tools to support project work. 

7. Prepare tutorial materials regarding the team project, the case study, team members’ responsibilities, team 
assignments and schedule. Include details of assignments, formats of project deliverables and final project 
report. 

8. Develop course website. 

Prepare teamwork related forms. 

9. Plan Project Review Exercise and reporting format. 

10. Plan team tutorial sessions. 

11. Make logistics arrangements for resources needed for team meetings. 
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TABLE 2 

ACTIVITIES IN THE PROJECT EXECUTION CATEGORY 



Project initiation 

Introduce the team concept and provide orientation regarding teamwork. 

1. Provide orientation regarding team functions in course website, project tools and project techniques. 

2. Review standards to be used in the project. 

3. Perform Student Background Evaluation Survey using SBE form. 

4. Organize students into teams (team size depends on the number of students in the class but should not 
exceed 5 persons). 

5. Identify team leaders based on student background survey; provide orientation to team leaders; make Team 
Composition Form available on course site. 

6. Identify the role players; provide orientation to role players. 

7. Take digital photograph of each team; post on course website. 

8. Introduce case study. Where ^propriate include presentation by industry consultant. 

9. Give orientation on team assignments to full class. 

10. Make example team project deliverables available to teams (reference only). 

Teamwork coordination 

1. Organize schedule of consultations with team leaders; with full team. 

2. Implement mechanisms for teamwork participation of all team members. 

3. Arrange availability of all project resources (computing, software tools, standards). 

4. Arrange project presentation schedule. 

Project consultation 

1. Hold weekly technical consultations with team leaders. 

2. Attend at least one team meeting per week with each team to consult on team-specific, technical issues. 

3. Provide role-specific orientation for role players of all teams, and follow-up during project. 

Project monitoring 

1. Monitor team leadership; take corrective action when necessary. 

2. Monitor team dynamics and progress; take corrective action whai necessary. 

3. Monitor all role players’ responsibilities and involvement in project. 

4. Monitor team collaboration on course site. 



team requirements. At a minimum faculty should review 
the principles of successful teamwork and teamwork 
methodology. A schedule should be arranged once the 
teams have been established for fiiture meetings with 
team leaders and their teams. The initiating aspect also 
includes orientation regarding the case study, the context 
and the scope of the team assignments, and other 
relevant topics. It has proved useful to make project 
folders from the archive available to teams for reference 
purposes, ensuring that such folders are not copied when 
referenced. These project folders have been stored in an 
archive as a record of past teamwork. 

The first team assignment focuses on teamwork 
planning following an project planning standard. For 
most of our team projects standard IEEE 1 05 8 (1998) 
was prescribed (refer also to Table 6). The importance 
of planning for quality as part of any project was also 



stressed by prescribing the standard for software quality 
assurance plans (IEEE730, 1998 in Table 6). 

Teamwork Coordination 

Coordination of teamwork involves the coordination of 
teamwork within teams, as well as coordinating work 
with a number of teams in the class. Coordination 
centers on logistics involving time, resources and 
location. Answers should be provided to the questions: 

• How is teamwork coordinated? 

• Do you appoint team leaders, or does the class choose 
them? 

• Do you assign roles for team members? 
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• Are teaching assistants involved? 

• How do you deal with culturally diverse teams? 

• How do you handle inadequate understanding within 
teams? 

• How do you handle dysfunctional teams? 

Faculty must work with team leaders to ensure that the 
necessary computing resources, relevant standards and 
software tools accessible to all members of the team. An 
equitable team consultation schedule should be followed 
(weekly meetings are appropriate), and opportunities 
should be provided for collabc^ation among teams. It 
has become evident that a fair amount of cross-training 
takes place within teams and also among different 
teams, all beneficial to the overall learning experience. 

Project Consultation 

Supporting the technical development work of a team 
project has proven to be most challenging. Some teams 
with skills and experience in the technical tasks of the 
project may need limited support. Such teams conduct 
productive team meetings and produce quality 
deliverables on time. They are usually keen to receive 
detailed feedback on deliverables and have a good 
teamwork dynamic and consequently high morale. The 
conditions for learning are clearly high in such team. 
Other teams struggle to come to terms with the context 
of team assignments for a case study, and to apply 
theoretical concepts and knowledge. For example, 
students may have studied a software design 
methodology, design processes and notations but still 
have significant problems to model a design for a given 
subsystem. If and when problems of this kind emerge 
faculty must increase the level of consultation and 
support for a team to ensure a successful team 
experience. 

Questions for this aspect include: 

• What issues are team-specific and what are of general 
concern? 

• How do you address such issues? (in class/on the 
website/ during consultation with teams) 

• How do you address responsibilities of role players in 
teams? 



• How do you coach team leaders? 

• How do you handle inadequate understanding within 
a team? 

Project Monitoring 

It is important to implement mechanisms for monitoring 
team performance. Although specific roles and 
responsibilities may have been assigned, the team 
assignments and the schedule reviewed, occasions 
invariably arise where team members fail to participate 
effectively. In some cases participation is irregular; 
deliverables are late, not according to specification or 
not submitted by the responsible team member at all. 
Since all teams have a quality assurance role late 
deliverables causes immediate frustration since the 
quality assuror does not have the opportunity to review 
the work before turning it in for grading. An issues like 
this one must be addressed as soon as it arises to avoid 
further conflict. 

In planning this aspect questions to be answered include : 

• How is team progress monitored? 

• How do you deal with team discipline? 

• How do you deal with dysfunctional teams? 

• How do you deal with a discontented team member? 

• Do you use groupware/Intemet/course website for 
virtual team monitoring? 

TEAM PROJECT EVALUATION 

One hopes to determine the outcomes of a team project 
in terms of the objectives for the course as a whole. 

Questions in this category ask: 

• How do you evaluate teamwork? 

• What percentage of the grade is based on teamwork? 

• What measures do you use? 

• How do you distinguish team member contributions? 
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Evaluation of teamwork usually focuses on the 
deliverables, but may also include evaluaticsi of the 
team process (Stephens and Myers, 2000) and team 
collaboration (McKeage et al, 1 999). These authors have 
proposed intervention mechanisms such as evaluating 
the team process to improve the effectiveness of student 
teams. We have used project review exercises also, 
where a team reviews deliverables of anottier team in 
the class, or where all teams review a specific 
deliverable from the project archive. Other mechanisms 
to determine individual accountability include team 
presentations, peer evaluations and participation during 
team tutorials. Table 3 summarizes typical activities in 
this category. 

TABLE 3 

ACTIVITIES IN THE TEAM PROJECT 
EVALUATION CATEGORY 



1 . Evaluate team deliverables. 

2. Evaluate revised team deliverables. 

3. Evaluate final project report. 

4. Evaluate project review exercise report. 

5 . Evaluate team progress meeting minutes (submitted 
by team leaders). 

6. Evaluate team member peer evaluations (submitted 
by each team member individually). 

7. Evaluate team member presentations. 

8. Evaluate overall team presentation. 

9. Assess each team’s experience (based on 
assessment by each team at the end of term). 

10. Assess project outcomes. 



TEAM PROCESS IMPROVEMENT 

Where teamwork forms part of a course it is desirable to 
establish a defined team process that represents a 
baseline according to which teamwork may be planned 
and performed. Defining the team process and 
documenting each team experience in a team process 
database make it possible to benefit from good ideas and 



techniques while avoiding past mistakes or negative 
outcomes. We need answers to the questions: 

• How do you document project outcomes? 

• How do you update your team process database? 

• What measure and metrics do you use? 

The ISO 15504 standard for software process 
improvement and capability determination (formerly 
known as SPICE) and CMMI framework which is an 
integration framework for mature IT (but mainly) 
software processes have proved useful as references to 
establish processes andacti vities for ttiis category. Table 
4 lists some typical activities in this category. 

TABLE 4 

TEAM PROCESS IMPROVEMENT CATEGORY 



1. Adopt an industry standard for team process 
improvement. 

2. Archive all A project reports. 

3. Archive all A project prototypes. 

4. Review teamwork approach (all phases) and analyze 
students’ responses to teamwork experience. 

5. Update team process database with project data. 



Guidelines regarding how to deal with differences in 
competence and skills among team members, how to 
deal with ethnically and culturally diverse team 
members, resolution of conflicts within teams, 
alternatives to teamwork in a project-oriented class are 
valuable for planning team-based, project-oriented 
courses. 

DISCUSSION 

The team-based, project-oriented approach described in 
this paper has matured over a number of years in 
project-based courses in software engineering, require- 
ments engineering, software design, database design, 
software process managanent, software quality 
assurance, system and information technology 
architectures, and software system metrics. Data on 34 
project-based classes has beenaccumulated representing 
teamwork by 150 teams. A comparative evaluation of 
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the team project outcomes of the courses in general, and 
of the deliverables of specific courses over a number of 
semesters has revealed a substantial improvement in 
quality. This quality improvanent is partially attributed 
to the structured project-oriented approach and the use 
of standardized notations, as well as to the software 
development methodologies that were adopted in the 
respective courses. 

Table 5 summarizes the standards and best practices 
used to guide teamwork in the listed courses. Projects 
were conducted by teams following a three stage 
process: project planning, project execution, and project 
review. Industry standards relevant to the tasks and 
activities of the three stages are provided in Table 5. 
Using the best practices embodied in these standards 
has provided faculty with a mechanism to introduce 
some measure of real-world exposure in the classroom. 
Feedback from graduates who are employed in the IT 
industry, though very positive, needs to be surveyed to 
quantify the outcomes of this team approach. 

As may be seen in Table 5 planning was a key category 
for all projects, requiring all teams to use IEEE Std 1058 
for planning the teamwork, supported also by the IEEE 
Std 730 for planning software quality assurance. For the 
Software Quality Management course the ISO9001 
standard was also used for planning the project work. 
For the course in Software Process Management the 



organizational life cycle processes of the ISO 12207 
Standard and the processes in the process improvement 
category of the IEEE Std 1074 were used for planning 
software process improvement projects. Project 
execution was supported by a range of standards 
dependent on the course subject. Project evaluation was 
guided by two standards among a number of other 
competing ones, namely ISO 15504 and IEEE Std 1028. 
The project managanent body of knowledge is 
documented in detail by PMI (2000) and in IEEE Std 
1490 (1998). Although this is a general project 
management reference the processes described there 
apply to IT projects also and are valuable for student 
projects. The Software Engineering Institute, Camegie- 
Mellon University has published a number of capability 
maturity models with key process areas in each of the 
team process categories discussed in this paper. 
Exposure to these models provides an additional 
advantage when conducting team projects for courses 
other than software process management. 

Further analyses of the team project data are planned to 
determine, for example), team performance versus 
individual performance of team members in a team, 
performance of teams in different sections of a 
particular course, individual performance of team 
leaders versus performance as team leaders of a team, 
and similar analyses. 



TABLE 5 

STANDARDS AND BEST PRACTICES ADOPTED FOR TEAM PROJECTS 



Courses 


Project Planning 


Project Execution 


Project Review 


Requirements Engineering 


IEEE730 


PMBOK 


ISO15504 




IEEE1058 


IEEE830 


IEEE 1028 


Software Design 


IEEE730 


PMBOK 


ISO15504 




IEEE1058 


IEEE1016 


IEEE 1028 


Database Design 


IEEE730 


PMBOK 


ISO15504 




IEEE1058 


IEEE1016 


IEEE 1028 


Software Process Management 


IEEE 1074 


IEEE1074 


ISO15504 




ISO12207 


ISO12207 


IEEE 1028 


Software Quality Management 


IEEE730 


IS09000 


ISO15504 




ISO9001 


ISO12207 


IEEE 1028 


Software/IT Architectures 


IEEE730 


IEEE1471 


ISO15504 




IEEE 105 8 


IEEE1220 


IEEE1028 


Software Metrics 


IEEE730 


IEEE1045 


ISO15504 




IEEE 1058 


IEEE1061 


IEEE 1028 
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TEACHING THE PROCESS OF 
REQUIREMENTS ANALYSIS: 
USING COMPARATIVE EXPERIMENTS 
TO INFORM TEACHING PRACTICE 
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ABSTRACT 

Requirements analysis has long been considered one of the most critical phases of the systems development life cycle. 
!t follows that requirements analysis should be an essential part of the training and education of future systems 
analysts. This paper identifies important skills associated with requirements analysis and then reviews the 
experimental literature to find out what can be used to inform our educational practice. The paper finishes with a 
set of recommendations for educators and suggestions for future research. 



INTRODUCTION 

Requirements analysis is one of the most important 
activities in the systems development process because it 
shapes everything that comes afterwards. 
Unfortunately, identifying complete and accurate 
requirements for an application is very difficult (Davis, 
1982). Given this importance and difficulty, it is 
obvious that teaching future systems analysts the 
knowledge and skills necessary to do requirements 
analysis properly is crucial for teachers and trainers in 
information technology. 

The purpose of this paper is to review the experimental 
literature concerning requirements analysis and to make 
recommendations on how the empirical findings can be 
used to improve the education of future systems 
analysts. The approach taken is to first associate cate- 
gories of knowledge/skills with the activities that make 
up requirements analysis. Thenexperimental studies are 
reviewed in light of these categories in an attempt to 
distill out results that can be applied in an educational 
setting. Finally, a set of recommendations are made 
based on these findings for educators and for 
researchers. 



REQUIREMENTS ANALYSIS 
KNOWLEDGE AND SKILLS 

Requirements analysis consists of the following 
activities (based on Nuseibeh & Easterbrook, 2000): (1) 
elicitation of requirements, (2) modeling and analysis 
based on elicited information, (3) communication of 
requirements and models to others, (4) validation of the 
accuracy of the requirements and models, (5) gaining 
agreement on the requirements by the affected parties, 
and (6) evolution of the requirements over the life of the 
application. To perform these activities systems analysts 
use a combination of knowledge and skills. Table 1 
shows how these activities map to categories of 
knowledge/skills. 

For the purposes of this paper, knowledge and skills are 
combined into a single category. Knowledge is the set 
of facts, rules, procedures, heuristics, and strategies that 
are learned and then applied to complete a particular 
requirements analysis activity. Skills are highly learned 
physical and cognitive behaviors. Knowledge ofhow to 
do something becomes a skill when it is practiced so 
much its execution becomes automatic, i.e., does not 
involve significant cognitive attention. The goal of 
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TABLE 1 

REQUIREMENTS ACTIVITIES 



Requirements 

Activity 


Description 


Activity Outcome 


Knowledge/Skills 

Required 


Elicitation 


obtaining requirements from users 


Analyst requirements 
understanding 


interpersonal; 

interviewing 


Analysis 


creating a conceptual model of 
current and desired systems 


Models 


modeling; 
problem solving 


Communication 


documenting requirements and 
communicating to others 


User/Manager 

requirements 

understanding 


documenting; 

comprehension; 

interpersonal 


Validation 


reviewing requirements for accuracy 
and completeness 


Assurance of requirements 
quality 


error finding 


Agreement 


gaining acceptance of requirements 
among stakeholders 


Agreement about 
requirements 


group processes; 
interpersonal 


Evolution 


updating requirements based on 
changing needs 


Up to date requirements 
documentation 


(all of the above) 



education is to transfer knowledge and develop skills 
that students need to be successful. For this paper we 
will use the term skills to refer to this combination of 
learning that produces effective performance. 

While there are many possible categories of skills, the 
ones used in this paper are described below. 

• Interpersonal: verbal and non-verbal behaviors 
involved in communicating with another individual. 
Listening is a key skill in this category. Knowledge of 
what behaviors should be exhibited is important but 
effectiveness usually requires that this knowledge be 
converted into skill. 

• Interviewing: high-level, goal driven cognitive 
behaviors designed to obtain specific information 
from another person. These skills focus on the 
structuring and high level processes of the interview. 
Lower level interpersonal skills are used to implement 
the structure and strategy. 

• Modeling: cognitive and physical behaviors required 
to create specificmodels. Higher level strategic skills 
that are applicable across different kinds of models are 
included in the problem solving category. 

• Problem solving: includes the cognitive behaviors 
associated with defining the problem the system is to 
solve, generating potential solutions, evaluating the 
solutions, and selecting one to recommend. 



Information to support problem solving is obtained by 
using interpersonal, interviewing, and modeling skills. 

• Documenting: verbal, physical, and cognitive 

behaviors involved with creating written, graphical, 
audio, video and/or computer materials that describe 
the requirements so that users, managers, or other 
analysts can understand them. Documentation often 
includes the products of modeling skills. 

• Comprehension: behaviors involved in reading and 
understanding models and requirements 
documentation. 

• Error-finding: cognitive behaviors associated with 
finding inaccuracies, inconsistencies, and 
incompleteness in documentation. 

• Group processes: verbal, non-verbal, and cognitive 
behaviors used to manage group processes. Group 
processes include group idea generation and 
evaluation, consensus building, negotiation, 
leadership, etc. Interpersonal skills may be important 
in group processes but are not included in this 
category. 

Two additional categories are mentioned here because, 
while not normally thought of as requirements analysis 
skills, they become important when the prototyping 
approach is used. 
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• Designing: cognitive behaviors used to create a 
technical design for an application that will meet the 
stated requiremaits of the user(s). 

• Constructing and testing: cognitive and physical 
behaviors used to create a working application based 
on the given design. 

The purpose of this paper is to review experiments that 
have compared techniques, methods, approaches, or 
levels of experience to find out what we have learned 
about th requirements analysis process that can inform 
our teaching. The comparative e?q)eriments that are 
reviewed for this paper are listed in Appendices A-F and 
described below. 

REVIEW OF EXPERIMENTS 

Twenty-seven experiments conducted over the last 25 
years are reviewed in this section. None of the reviewed 
experiments was designed as a learning study. Most 
were meant to prove that one technique, method, or 
approach was better than another. In spite of this, there 
are aspects of the research that can inform our teaching. 
Studies that compare high rated analysts to novice 
analysts can suggest skills that should be developed in 
our students. Studies that teach subjects the techniques, 
methods, or approaches that are compared can tell us 
which approaches are easier to teach and which are more 
effective when taught. Since the focus here is on 
developing necessary skills for successful requirements 
analysis, we will review the research in terms of the key 
skills listed in Table 1. 

Interpersonal Skills 

Interpersonal skills have long been regarded as essential 
for successful systems analysts (Leitheiser, 1992). 
Skills include the ability to question, listen, empathize, 
reflect, paraphrase, clarify, challenge and summarize. 
These skills represent the micro skills of effective 
communication. Higher level skills involving the 
planning and overall structuring of interviews will be 
dealt with in the interviewing skills category. Micro 
skills make up part of the “effective communication 
patterns” called for by some researchers (Bostrom & 
Thomas, 1983; Bostrom, 1984; Guinan & Bostrom, 
1986). Unfortunately, no experiments comparing 
training or techniques in this area were found. One 
study (Tan, 1992) examined how three pre-existing 
interpersonal skills; i.e., orientation toward users, 
management of the interview, and building of rapport. 



impacted the mutual understanding of systems 
requirements (see Appendix A). Both verbal and 
nonverbal behaviors were examined. The results were 
mixed with one measure of client orientation, two 
measures of interview management, and one measure of 
rapport found to be related to at least one measure of 
mutual understanding. Given the paucity of research in 
this area, it is clear that there is much more to be learned 
about how interpersonal skills are related to 
requirements analysis success. Tan’s results suggest 
that educators should at least be aware of the possible 
impact of these often neglected skills. 

Interviewing Skills 

Interviewing skills are those planning and structuring 
skills that are used by the analyst to organize and 
manage requirements analysis interviews. The structure 
of the interview is usually based on some goals and a 
plan for how to achieve the goals. For example, 
decision analysis has the goal of identifying the 
information needs for a user’s decisions. A plan is 
created to define decisions and then for each decision, 
identify required information. This plan establishes the 
interview structure and the content of the question 
prompts. At the prompt level, the micro interpersonal 
skills mentioned above take over. 

A number of studies (Appendix B) compare different 
approaches to structuring the interview process. 
Structuring approaches that have been studied focus on: 

• decisions (Munro& Davis, 1977;Teng& Sethi, 1990; 
Agarwal & Tanniru, 1990), 

• existing data (Munro & Davis, 1977; Teng & Sethi, 
1990), 

• prototypes (Alavi, 1984; Boehm, et al., 1984;Teng & 
Sethi, 1990), 

• goals (Zmud, et al., 1993; Browne & Rogich, 2001), 

• Critical Success Factors (Zmud, et al., 1993), 

• imagery (Zmud, et al., 1993), 

• episodic knowledge (Moody, et al., 1998), 

• process inquiry (Marakas & Elam, 1998), 

• checklists (Mo<xe, Shipman, 2000), 
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• computer support (Moore, Shipman, 2000), 

• task characteristics and requirements obstacles 
(Browne & Rogich, 2001), and 

• interrogatories (Browne & Rogich, 2001). 

Sometimes the alternative to the experimental treatment 
was an untrained control (e.g., Agarwal & Tanniru, 
1990; Marakas &Elam, 1998; Moody, et al., 1998). 

What can academics learn from these studies? First, 
structure seems to matter. In those studies that 
compared a structured interviewing approach to an 
unstructured one, the structured approach produced 
more/better requirements or knowledge (Agarwal & 
Tanniru, 1990; Moody, et al., 1998; Marakas & Elam, 
1998). Second, there are mixed results about which 
structured approach to take. Based on direct 
comparisons the following conclusions can be made. 

• Decision analysis may be better than data analysis for 
programmed decisions (Munro 8c Davis, 1977), 

• Prototyping may produce better requirements and 
more satisfied users than other approaches (Alavi, 
1984; Teng& Sethi, 1990), 

• Using positive imagery may result in more 
information requirements than using a goal focused 
approach (Zmud, et al., 1993), 

• A questionnaire based approach can produce more 
functional requirements and may lead to more creative 
solutions than a computer tool approach (Moore & 
Shipman, 2000), 

• A computer tool approach may lead to more 
procedural requirements and more documentation than 
a questionnaire approach (Moore & Shipman, 2000), 
and 

• A task characteristics approach derived from a 
cognitively based model may produce more 
requirements than other structured alternatives 
(Browne & Rogich, 2001). 

Hopefully in the future more head to head comparisons 
will be done to he^ sort out which structure is the most 
effective one to teach to our students. 



A special note should be made about prototyping. 
Prototyping is listed under interviewing skills when user 
interaction is part of the process. The reason for this is 
that the prototype defines the structure of the initial and 
follow-up interviews. In our review it also shows up 
under problem solving skills because analysts must 
create a prototype to solve their users’ problems. Other 
skills are also involved, many of vdiich are not usually 
associated with information requirements analysis. 
Analysts must be able to design, construct and test 
prototypes. If they have poor skills in these areas the 
requirements analysis process may be hindered. On the 
other hand, good design and construction skills may 
help them overcome deficiencies in interpersonal and 
interviewing skills. Future research will hopefully help 
us tease apart the effects of different skills on the 
success of prototyping. 

One interesting, and troubling, observation from the 
studies is the finding that prior experience did not 
produce higher performance. Agarwal and Tanniru 
(1990) and Marakas and Elam (1998) both found that 
systems analysts did no better than students in their 
experiments. Is this an artifact of the experimental 
design or is it a generalizable result? 

Modeling 

The studies comparing different modeling approaches 
(Appendix C) were designed to show which approaches 
are better than the others. They also suggest which can 
be more effectively taught. The modeling approaches 
differ primarily in their emphasis; i.e., (1) on processes, 
(2) on functions, (3) on data, or (4) on objects and 
classes. All but one of the studies in involve process 
modeling so we will use it as the basis for summarizing 
results. The major conclusions that can be drawn are: 

• process modeling is easier to learn and use than 
functional modeling (Yadav, et al., 1988), 

• process modeling produces better results on process- 
oriented tasks and elements than object-oriented 
(Agarwal, et al., 1996), 

• process modeling may result in less complete models 
and may be viewed as harder to use than object- 
oriented modeling (Wang, 1996a), 
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• process modeling may result in better model 
semantics than object-oriented modeling (Wang, 
1996b), 

• structured interviewing results in better process 
models than unstructured interviewing (Marakas & 
Elam, 1998), 

• process modeling may result in lower cognitive 
workload than object-oriented modeling (Morris, et 
al., 1999), and 

• object-oriented modeling is preferred over process 
modeling by inexperienced subjects who feel it is 
easier to use (Morris, et al., 1999) — although using it 
produced no improvement in performance. 

In the one study that did not involve process modeling, 
Kim and March (1995) found that analysts captured 
semantics better with an entity-relationship mode! than 
with a binary object model. They also felt that ER was 
easier to use and more valuable. 

One obvious conclusion is that process modeling should 
not be abandoned for strictly static object-oriented 
approaches. In fact, popular object-oriented develop- 
ment processes include an array of process focused 
models that extend the standard class and object 
representations that are the basis of object-oriented 
approaches. For example, the Unified Software De- 
velopment Process (Jacobson, Booch, and Rumbaugh, 
1999) includes use-case modeling, activity diagrams, 
collaboration diagrams, statechart diagrams and 
sequence diagrams to capture various parts of the 
processes being modeled. It seems clear that process is 
important but given the popularity of object-oriented 
methods we should move on to experiments that 
examine these alternative approaches to capturing 
process information. 

Problem Solving 

The studies of problem solving were of two types 
(Appendix D). The first examined the abstract cognitive 
problem solving process while the second was 
concerned with the concrete process of creating a system 
solution. Included in the first category are two similar 
studies on specific problem solving behaviors. A 
comparison of their results can be summarized as: 

• Hypotheses: high rated analysts discard mere 
hypotheses (Vitalari & Dickson, 1983) vs. no 



differences in the total number of hypotheses stated, 
but experts tested and discarded more of them 
(Schenk, et aL, 1998), 

• Strategies: high rated analysts modified and 

verbalized more strategies (Vitalari & Dickson, 1983) 
vs. novices verbalized more strategies than experts but 
they tended to be more “text book” (Schenk, et al., 
1998), 

• Triggers: high rated analysts search fa* more triggers 
(Schenk, et al., 1998) vs. no differences in 
verbalization and searching for triggers except that 
novices tended to be less specific than experts 
(Schenk, et aL, 1998), 

• Goals: high rated analysts tended to set more goals 
(Vitalari & Dickson, 1983) vs. experts verbalized 
more goals than novices (Schenk, et al., 1998), 

• Heuristics: high rated analysts apply more heuristics 
(Vitalari & Dickson, 1983) vs. neither experts nor 
novices verbalized many heuristics (Schenk, et al., 
1998). 

The latter study also found that: (1) experts took longer 
on the task; (2) experts referenced more “domain 
specific” issues and covered more kinds of domain 
issues; (3) novices referenced user involvement more 
but experts mentioned it more consistently throughout 
the task; (4) experts focused on functional requirements, 
types of reports, and procedures while novices focused 
on development issues, design of system, and purposeof 
system; and (5) experts’ approach was bottom-up while 
novices was top-down (Schenk, et al., 1998). 

It is clear that there are differences between analysts in 
the way they solve problems. The results suggest skills 
that novices should do to be more like expert analysts, 
but they do not tell us how to achieve them with 
education and training. An experiment comparing how 
different training treatments impact the development of 
these skills would help us take advantage of these 
interesting findings. 

Another study of cognitive problem solving did examine 
the impact of treatment conditions on problem solving. 
V essey and Conger ( 1 993 & 1 994) found that an obj ect- 
oriented approach resulted in more problem solving 
breakdowns than process oriented or data oriented 
approaches. They also found that problem solving in an 
unfamiliar domain resulted in both longer modeling time 
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and more unresolved problem solving breakdowns. In 
addition to contributing to the argument mentioned 
above about process modeling, the findings also suggest 
that there may be a reason for students to specialize in a 
functional area, industry or type of application. 

Two prototyping studies are included in problem solving 
because subjects went beyond identifying requirements 
to building solutions to users’ problems. This process 
includes problem solving as well as design, 
construction, and testing. One of the studies involved 
interviewing (Alavi, 1984) while the other did not 
(Boehm, et al., 1984). If the outcomes of the proto- 
typing studies can be viewed as the ability to solve 
users’ problems then the results suggest that 

• prototyping leads to higher user satisfaction than 
traditional approaches (Alavi, 1984), 

• prototyping takes less time and results in a smaller 
system (Boehm, et al., 1984), 

• prototyping results in a system with less functionality 
and less robustness (Boehm, et al., 1984), and 

• prototyping results in a system that is easier to use and 
maintain (Boehm, et al., 1984). 

As mentioned above, prototyping involves a range of 
skills beyond problem solving. 

Documentation and Comprehension 

Documentation and Comprehension are opposing sets of 
skills. Documentation is the ability to create under- 
standable representations of requirements while 
comprehension is the ability to understand them. Only 
one of the reviewed studies (Appendix E) focused on 
comprehension. Agarwal, et al. (1999) found that 
students who used process-oriented representations 
better understood aspects of the model that involved 
combinations of process and structure than did students 
using an object-oriented representation. There were no 
differences on pure structure or pure process questions. 
Kim and March (1995) did have a comprehension 
component in their experiment but did not find any 
significant differences between their modeling groups 
on this measure. 



Error Finding 

Error finding is the ability to find problems in the 
comprehended documentation and is essential to 
validation. Results from the three reviewed studies that 
examined error finding (Appendix F) can be summed up 
as: 

• different teams find different errors, so having 
multiple teams looking at the same documentation can 
be beneficial (Schneider, etal., 1992), 

• students who reviewed entity relationship models felt 
that they were more valuable than students reviewing 
binary object models, even though no significant 
performance differences were observed (Kim & 
March, 1995), 

• a structured error finding approach that divides and 
focuses efforts among team members results in more 
identified problems than approaches that are less 
structured and focused (Porter, et al., 1995), and 

• using a group session to identify errors after 
individual reviews may not result in a net increase in 
the number of errors identified (Porter, et al., 1995). 

As was observed in the Interviewing section, more 
structure seems to improve the performance in this 
requirements analysis activity. It may not be clear yet, 
what structures are best, but we can conclude that we 
should give students a detailed approach for error 
finding. 

Group Processes 

Group processes are in effect for any study that involves 
teams. For example, the Schneider, et al. (1992) error 
finding study reviewed above used teams to find faults 
in documentation. These studies are not reviewed here 
because the authors did not focus on the group processes 
used so we cannot make any conclusions about 
associated skills. In fact, group processes involved in 
requirements analysis have been little studied. The 
experiments discussed here (Appendix G) were mostly 
concerned with the impact of group technology on these 
processes. Two related experiments (Ocker, etal., 1996 
and Ocker, et al., 1998) and one other experiment 
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(Damian, et al., 2000) compared various group settings, 
some supported by group support technology and some 
not. The authors used the face to face setting as a 
control and tried to find GSS technologies that provide 
improvement in requirements analysis processes. 
Summarizing the findings produces the following. 

• asynchronous group support technology resulted in 
more creative and better requirements than face to 
face settings (Ocker, et al., 1996), 

• a combined approach using asynchronous group 
support and face to face meetings produced more 
creative and better requirements than either approach 
by itself (Ocka*, et al., 1998), 

• a configuration with the analyst and one user in one 
location, video conferencing with a facilitator and 
second user in another location, resulted in better 
negotiated requirements (Damian, et al., 2000), and 

• local participants were regarded as more emotional, 
more argumentative, and more competitive than 
remote participants (Damian, etal., 2000). 

The results say more about the possible use of 
technology than about what group process skills should 
be taught. The finding that location and technology may 
affect how team members are perceived (Damian, et al., 
2000) deserves further exploration and may effect the 
training of analysts. Clearly, more research needs to be 
done on group processes in requirements analysis, both 
computer supported and unsupported. 

CONCLUSIONS 

This paper reviewed comparative experiments that 
investigate requirements analysis activities. The goal of 
the paper is to derive suggestions for how to improve the 
teaching of requirements analysis skills for future 
systems analysts. One major conclusion is that there is 
a lot yet to be learned if we are to base our education on 
experimental results. Hopefully, this review has helped 
to identify opportunities for researchers so that this need 
can be met. The following recommendations are based 
on the review of experiments and organized by 
requirements analysis activities. 

Elicitation 

The research shows that elicitation success can be 
enhanced by teaching a specific process for structuring 



requirements interviews. There are many options for 
this structuring but the task characteristics approach 
created by Browne and Rogich (2001) is particulariy 
interesting and deserves more exploration. There is also 
evidence that improving the interpersonal skills of 
analysts will benefit elicitation. Teaching students how 
to build rapport with users, develop a client orientation, 
and manage the interview process should improve their 
results. 

A consistent finding is that prototyping may provide 
benefits for requirements analysis in some setting^. This 
suggests that prototyping should be included in our 
education process. A challenge for prototyping, how- 
ever, is that it involves a large array of knowledge and 
skills that go beyond what is traditionally considered as 
necessary for requirements analysis. Questions about 
the importance of design, construction, and testing skills 
in determining requirements through prototyping have 
yet to be answered. 

Analysis 

Analysis results are improved by effective models and 
good modeling techniques. The research suggests that 
standard class and object models do not do a good job of 
capturing required system behaviors. Since object- 
oriented methods are becoming the standard way to 
teach systems analysis, instructors should be aware of 
this deficiency. 00-based methods for effectively 
capturing behavior (e.g., use cases, sequence diagrams, 
activity diagrams, statechart diagrams, and collaboration 
diagrams) exist but have yet to be tested experimentally. 

Future analysts can be helped if they can be taught more 
effective problem solving techniques including (1) 
setting goals, (2) developing solution strategies, (3) 
recognizing triggers in problem, (4) generating, testing 
and discarding of hypotheses, and (5) using heuristics. 
Unfortunately, specific methods for training students in 
these techniques are not widely available. 

Communication 

Communication to users and manages is helped by 
documentation and models that are easily compre- 
hended. The research in this area tells us that class and 
object models by themselves do not do a good job of 
communicating behavior. This activity would also 
benefit from additional research into the effectiveness of 
00-based behavioral representations. 
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Validation 

The validation activity benefits from a structured 
approach to error finding. The structured approach 
described by Porter, et al.(1995) seems to have promise 
and should be compared to other, equally structured 
methods. Students should also be taught to bring 
additional perspectives to the error finding process. 
Research tells us that more participants mean more 
errors are found. 

Agreement 

Gaining agreement on system requirements is a group 
process. No research was found comparing different 
face to face situations but there is research comparing 
various computer mediated processes with face to face 
settings. Some use of group support systems in the 
process seems to increase the creativity and quality of 
the requirements produced. The specificcombination is 
not clear but educators may want to explore using GSS 
as part of their requirements training. 

Evolution 

No new skills were designated for the evolution activity. 
No experimental studies were directed specifically at it. 
The ongoing evolution of requirements is an area that 
has not received much attention from researchers or 
educators. 

In conclusion, a significant number of experiments have 
been done that investigate the requirements analysis 
process but more are needed. This paper has sum- 
marized many of these studies and has tried to glean 
from them lessons that can be applied to educating 
future systems analysts. It also points out areas where 
more work needs to be done and invites researchers to 
fill in the knowledge gaps. 
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APPENDIX A 

INTERPERSONAL KNOWLEDGE/SKILLS 



Study 


Learning Treatment 


Subjects 


Results 


Tan, 92 


none 


28 systems analyst/user pairs 


comprehension is related to client orientation, 
interview management and rapport on some 








measures 





APPENDIX B 

INTERVIEWING KNOWLEDGE/SKILLS 


Study 


Learning Treatment 


Subjects 


Results 


Munro, 


none (experimenter 


experimenter as analyst; 16 


for programmed decisions, decision analysis 


Davis, 77 


applied data or 
decision analysis) 


college administrators 


produced better requirements 


Alavi, 84 


traditional life c ycle 
vs. prototyping 


29 MBA’s in 9 analyst 
groups; 34 MBA’s in 9 user 
groups 


prototyping lead to higher user satisfaction with 
system and process 


Agarwal, 

Tanniru, 

90 


structured vs. 
unstructured 
interviews; novice vs. 
experienced 


MBA ’s and systems ana lysts 


structured > unstmctured on acquiring knowledge; 
experienced vs. novice - no difference 


Teng, 


decision analysis vs. 


12 MBA users (4/case), 12 IT 


prototypers produced higher quality requirements; 


Sethi, 90 


data analysis vs. 
prototyping 


PhD analysts (4/method) 


users & analysts preferred prototyping 


Zmud, et 


goals oriented vs. CSF 


4 PhD students (I/method) 


positive imagery produced more information items 


aL, 93 


method vs. positive 
imagery vs. negative 
imagery 


and 100 business senior 
students (25/method) 


than goal oriented 


Moody, et 
al„ 98 


standard vs. cognitive 
interviewing 


10 senior students, 42 
librarian users 


cognitive interviews produced more events 


Marakas, 


semantic structured vs 


20 IT students and 20 


semantic subjects had better DFD models; 


Elam, 98 


unstructured 
interviewing; 
novice vs. experienced 


systems analysts 


experience - no difference 


Moore, 


GUI design tool vs. 


4 students with questionnaire; 


questionnaire produced more functional 


Shipman, 

00 


questionnaire 


6 with tool 


requirements and more creative requirements; 
tool produced more procedural requirements and 
more text 


Browne, 


task characteristics 


15 employee subjects each 


task char, prompting produced more requirements. 


Rogich, 

01 


prompting vs. 
interrogatory 
prompting vs. 
semantic prompting 


treatment 


more process requirements, more information 
requiremen ts. 
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APPENDIX C 
MODELING SKILLS 



Study 


Learning Treatment 


Subjects 


Results 


Yadav, et 


process vs. functional 


20 grad students in 10 teams 


process modelers felt method was easier to learn 


al., 88 


modeling 


(5 per treatm ent) 


syntax and to draw. 


Kim, 
March 95 


entity relationship vs. 
binary object modeling 


Exp, 2: 26 analysts 


Exp, 2: analysts did better on semantics with ER 
and felt ER was easier and more valuable; 


Agarwal, 


process vs. object- 


24 business students 00 


process modeling produced better results on 


et al,, 96 


oriented modeling; 
process-oriented vs. 
structure-oriented 
tasks 


method; 19 business students 
process method 


process-oriented task and on all process elements 


Wang, 96 


process vs. object- 
oriented modeling 


32 undergrads do both 
methods 


00 method resulted in more complete models, and 
was rated easier to use and better overall 


Wang, 

96b 


process-oriented vs. 

object-oriented 

modeling 


44 IT undergrads (24 process 
and 20 00) 


process subjects scored higher on semantics 


Marakas, 


semantic structured vs 


20 IT students and 20 


semantic subjects had better DFD models; 


Elam, 98 


unstructured 
interviewing; 
novice vs. experienced 


systems analysts 


experience - no difference 


Morris, et 


process-oriented vs. 


34 IT majors (experienced) 


all subjects reported lower cognitive workload 


al„ 99 


object-oriented; 
prior exp erience w ith 
process-oriented 
modeling 


and 37 pre-business students 


with process-oriented method; experienced 
subjects had better solutions; inexperienced 
subjects prefeired 00 and felt it was easier 





APPENDIX D 

PROBLEM SOLVING KNOWLEDGE/SKILLS 


Study 


Learning Treatment 


Subjects 


Results 


Vitalari, 

Dickson, 

83 

Alavi, 84 


high rated vs. low 
rated analyst 


systems analysts 


problem solving differences exist 


traditional life cycle 
vs. prototyping 


29 MBA’s in 9 analyst 
groups; 34 MBA’s in 9 user 
groups 


prototyping lead to higher user satisfaction with 
system and process 


Boehm, et 


specifying approach 


1 1 grad students in 4 


prototyping took less time and produced a smaller 


al., 84 


vs. prototyping 


specifying teams; 7 grad 
students in 3 prototyping 
teams 


system; 

prototype had less functionality and robustness but 
was easierto learn and maintain. 


Vessey, 
Conger, 
93 & 94 


process-oriented vs. 
data-oriented vs. 
object-oriented 
modeling; 

familiar vs. unfamiliar 
doma in 


6 IT students (2/method) 


object-oriented subjects had more problem solving 
breakdow ns; 

unfamiliar domain required more time and resulted 
in more unresolved breakdowns 


Schenk, et 
al,, 98 


novices vs. low rated 
analysts vs. high rated 
analysts 


7 novices (< 6 mos, 
experience), 9 low rated 
experienced, 9 high rated 
experienced 


differences in problem solving behaviors 
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APPENDIX E 

DOCUMENTATION & COMPREHENSION KNOWLEDGE/ SKILLS 



Study 


Learning Treatment 


Subjects 


Results 


Kim, 


entity relationship vs. 


Exp, 1 : 28 grad students; 


Exp. 1: no differences in comprehension; 


March 95 


binary object modeling 






Agarwal, 


process-oriented vs. 


Exp, 1: undergrads (18 00 


process-oriented subjects did better on combined 


et al., 99 


object-oriented; 


and 18 PO); 


process/structure questions 




different case for each 


Exp. 2: undergrads (18 00 






experiment 


and 17P0) 





APPENDIX F 

ERROR FINDING KNOWLEDGE/SKILLS 


Study 


Learning Treatment 


Subjects 


Results 


Schneider, 


none 


computer science students in 


different teams found different errors 


et al., 92 




teams 




Kim, 


entity relationship vs. 


Exp. 1: 28 grad students; 


Exp. 1: students felt ER more valuable; 


March 95 


binary object modeling 






Porter, et 


ad hoc vs. checklist vs. 


48 CS grad students in 3 


scenario method > ad hoc, checklist; 


al., 95 


scenario fault 


person teams 


groups did not produce more faults than 




detection methods 




individuals 





APPENDIX G 

GROUP PROCESS KNOWLEDGE/SKILLS 


Study 


Learning Treatment 


Subjects 


Results 


Ocker, et 


GSS support vs. face- 


214 IT and MBA grad 


GSS more creative and better requirements; 


al., 96 


to-face; 
structured vs. 
unstructured process 


students in 40 teams 


structure - no difference 


Ocker, et 


GSS synchronous vs. 


teams from Ocker, et al., 96 


combined was more creative and produced better 


al., 98 


GSS asynchronous vs. 
face-to-face vs. 
combined f-to-f & 
asynchronous 


and 22 teams of IT students 


requirem ents 


Damian, 


face-to-face and video 


15 student teams with 1 


analyst & user conferencing to facilitator and user 


et al., 00 


conferencing 

combinations 


analyst and 2 users 


worked best 
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STRATEGIC PHILANTHROPY 
ON UNIVERSITY CAMPUSES: 
HOW DOES IT AFFECT CURRICULUM? 



Camille Fran^oise Rogers 
Georgia Southern University 

Thomas L. Case 
Georgia Southern University 



INTRODUCTION 

Economic need has placed many colleges and 
universities at the mercy of businesses that have 
enormous pocketbooks and overpriced products. 
Software vendors (and some hardware vendors) are now 
playing a role in the IS/IT curricula at numerous 
universities via product donations and partnership 
programs. Microsoft, SAP, and other mqor vendors 
have forged education partnerships with a large number 
of universities and have enabled students to gain 
hands-on experience with tools and applications that 
have the potential to strengthen and differentiate the 
skill sets they possess upon completing their degrees. 
There is, however, a darker side to these partnerships 
that many IS/IT professors and the administrators to 
whom they report often choose to overlook and 
downplay. Such partnerships, for example, include not 
so subtle marketing of vendor products to students as 
well as exploiting economic realities faced by colleges 
and universities by exchanging product donations for the 
ability to advertise to a captive audience. 

Today, access to current technology is important at all 
levels of education. Hardware, software, and training 
of instructors are typically critical determinants of 
success in today’s classrooms. IS/IT technology 
currency is an ongoing challenge for educational 
institutions. This is often very expensive to maintain and 
the amounts included in administrative budgets to do so 
often falls woefully short of the mark. In some 
instances, computers are obsolete by the time they are 



installed and maintaining hardware currency requires 
their replacement within two or three years. In addition, 
because the technology (both hardware and software) is 
so dynamic, it is difficult for educators to abreast of the 
changes. Funding and other necessary support for 
ongoing faculty devclq^ment is typically under-funded, 
if it is funded at all. 

Any serious commitmait to maintaining currency in 
technology infrastructure requires a steady stream of 
cash. Given the economic realities facing higher 
education funding (both public and private) in many 
states, many universities have turned to technology 
vendors for assistance in the form of donations, 
sponsorships, and full-blown education partnership 
programs. In general, vendors have been responsive. 
They have provided needed software (and/or hardware) 
and have provided instructors with the training they 
need to leverage upgrades and other new developments 
for the benefit of their students. In a previous study a 
list of partnership opportunities between industry and 
academia were developed and summarized as shown in 
Table 1 (Duncan, 2000, 98). 

In some instances, major employers of a university’s 
IS/IT graduates have helped the university secure 
vendor donations. This assistance is often inspired by 
their desire for new hires to have specific skill sets. 
Despite the recent economic downturn, the Information 
TechnologyAssociation of America (ITAA) reports that 
1.1 million new IS/IT jobs will be created in 2002 and 
that more than half of these (600,000) will go unfilled 
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TABLE 1 



• Advisory Committees 

• Student Internships 

• Industry Projects for College Seniors 

• Faculty Intemships/Consulting 

• Faculty/Industry Exchange Programs 

• Faculty Training and Updating 

• Hardware/Software Resource Sharing 

• Mentoring 

• Research Grants and Contracts 



because of a lack of qualified workers (ITAA Press 
Release, 2002). While a number of IS/IT workers 
displaced as a result of the economic downturn may 
argue otherwise, the ITAA study suggests that 
businesses across the nation continue to face a shortage 
of appropriately skilled technology workers. And, the 
outlook for the future does not look any less bleak. 

The state of Georgia’s attempt to position itself for the 
new economy faces similar challenges. According to a 
report released by the U.S. Department of Commerce 
there will be an average shortage of 5,560 technology 
workers every year until 2006 in the state of Georgia 
(1999). Collectively, however, the state’s public 
universities and technical schools graduate less than 
2500 IS/IT students each year. In order to respond to 
this gap, the University System of Georgia is taking 
steps to increase the number of graduates in technology 
fields such as computer programming, computer 
engineering, systems analysis, and Web development. A 
high-visibility aspect of the University System’s attempt 
to close the gap is the creation of a new School/College 
of Information Technology at Georgia Southern 
University (GSU). This project, backed by Governor 
Roy Barnes, includes a $33 million building. According 
to one group of researchers this “will make the coastal 
region of Georgia and the state as a whole, an attractive 
site for high-tech relocation and expansion” (Bums et. 
al, 2000, pg. 287). The school/college is being tasked 
by the state to more than triple the number of IS, IT, 
computer science, and computer engineering students 
graduating annually (to approximately 500). While this, 
by no means, will close the gap between the number of 
IS/IT graduates needed by businesses in the state and the 
number produced by the state’s publicly funded schools, 
it does represent a step in the right direction. 

There are many issues that universities should consider 
before accepting vendor donatiais, entering vendor- 



sponsored partnership programs, or allowing advisory 
boards to have a direct say in curriculum decisions. 
These issues have been previously outlined by other 
researchers and we don’t pretend to be saying anything 
that is new or earth-shattering in the pages that follow. 
However, we do feel that these are issues that we need 
to remind ourselves about and not dismiss out of 
expediency. These issues will be surfaced as we look at 
how a series of donations and partnerships has shaped 
the IS curriculum at Georgia Southern University. Key 
issues in our discussion include who is providing the 
resources, what strings are attached to donations and 
partnership programs, and what is the long-term impact 
of donations/partnerships on IS/IT curricula. Perhaps 
the most important issue is whether donations/ 
partnerships result in a mutually beneficial collaboration 
or if there is a decided imbalance that favors the 
providers (vendors). We have included a series of 
photos to assist in illustrating the impact that donations/ 
partnerships have had on the College of Business 
Administration at Georgia Southern University. 

WHO IS PROVIDING RESOURCES? 

The bulk of the additional funding (beyond state 
appropriations and the university’s student technology 
fee) for technology at Georgia Southern comes from 
three primary sources, all of whom have an interest in 
having access to highly skilled college graduates. The 
philanthropists can be categorized as technology- 
producing companies (such as SAP America), 
non-technology corporations who have technology 
needs (such as United Parcel Service), and various 
government sources. For example, technology funding 
at the university is also being provided by the Georgia 
Lottery program implemented in 1993 through 
distributions made to the Board of Regents (Dolan et.al, 
1997). Other potential sources for government funding 
for IS/IT programs include the national Tech Corps 
program. Both legislators and the governor have helped 
ensure funding for the new school/college of 
Information Technology at Georgia Southern through 
budget appropriation processes. 

As noted previously, COB A’ s School of Information 
Technology (IT) is being positioned to eventually 
blossom into a College of IT. Being initially housed in 
COBA has meant that many of the mechanisms 
established to provide external support and involvement 
in COBA have been carried over to the School of IT, 
including having corporate executives serve on an 
advisory board. 
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The advisory board for COB A includes executives from 
Georgia Pacific, BellSouth, Georgia Power, and 
Gulfstream Aerospace Corporation (see photo), just to 
name a few. When the current business building 
opened its doors in 1995, the Dean of COBA wanted to 
express appreciation for the many corporations who 
provided funding. As a result for a certain level of 
donation they could have a faculty office, classroom, 
computer lab, or common area named in their honor as 
seen in some of the photos throughout this text. 

FIGURE 1 

GULFSTREAM AEROSPACE CORPORATION 



Gulfstream Aerospace, the Southern Company, and 
United Parcel Service. 

RESOURCES PROVIDED 

In addition to financial donatiais, corporations are 
providing many resources to help support technology 
programs at universities. These include scholarships, 
internships, assistance in securing hardware and 
software donations, and much more. Representatives 
from businesses also give of their time and serve on 
advisory council committees. Some even provide faculty 
development training in current technology. 





GRADUATE 

STUDIES 




IN APPRECIATION OF GENEROUS 
SUPPORT THIS OFFICE COMPLEX 
IS NAMED IN HONOR OF 
GULFSTREAM AEROSPACE 
CORPORATION 



A high-powered Board of Advisors has also been 
assembled to help ensure the success of the new 
School/College of Information Technology. It consists 
of executives from a wide-range of technology 
producers and heavy tedmology user organizations 
including BravePoint, BellSouth, Clarkston Potomac, 
Coca-Cola, CommerceQuest, Georgia-Pacific, SAP 
America, Southern Company, TSYS, United Parcel 
Service, and Verisign,. Several of the companies that 
represented on COBA’s advisory Board are also 
represented on that for the School of IT. The school’s 
advisory board also includes the Commissioner of the 
Georgia Department of Industry, Trade, and Tourism as 
well as the state’s QO and executive director of the 
Georgia Technology Authority. The schools board of 
advisors includes representatives from all three of the 
major interest groups mentioned earlier: 
technology-producers, heavy technology-users, and state 
government. Some of the tedinology-producers on the 
board include BravePoint, SAP, Verisign, Commerce- 
Quest, and Safe Systems, Inc. Heavy technology users 
include BellSouth, Coca-Cola, Georgia Power, 



GSU’s Lotus Donation 

As an example of technology donations, Lotus 
Development Corporation provided a $1 million 
software site license in 1994 to the university that 
allowed any computo" on campus to have the Lotus 
SmartSuite software program installed on it. In 
addition, the university also received a site license for 
the Lotus Notes which could be leveraged to facilitate 
communication among administrators and to create an 
intranet. The $1 million software donation from the 
Lotus Development Corporation to the College of 
Business Administration (COBA) also qualified the 
college for matching lottery money. Needless to say, this 
was a powerful incentive to accept this vendor’s 
donation, even if though there were strings attached. 

The Lotus donation is a good example of strategic 
philanthropy. It gave the company visibility at the 
university that we would not have gotten otherwise. For 
example, for their generosity, they got a sponsorshp 
plaque on a computer lab, which serves as constant 
reminder of their donation forstudents, faculty andstaff. 
In addition, the Lotus SmartSuite software program was 
integrated into multiple courses across the business 
curriculum including, but not limited to, Introduction to 
Computers, Business Statistics, Accounting, and 
Finance courses. Such integration was one of the major 
strings attached to the donation. It spawned a great deal 
of debate, largely because of competing (Microsoft) 
products with larger market shares. The lack of Smart- 
Suite lab manuals for the Introduction to Computers 
course was also a bone of contention until several 
faculty members (including one of the authors of this 
paper) published a series of books to address this need. 

Lotus Development Corporation, now part of IBM, got 
a great deal of mileage out of its donation. It succeeded 
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in having the SmartSuite installed in more than 1000 
computers across campus and in having thousands of 
students exposed to this product via the Introduction to 
Computers and the other courses in which it was used. 
By doing so, the company also succeeded in 
demonstrating that its products could be integrated into 
the business curriculum. And, the plaque recognizing 
the company’s generosity still hangs outside COBA’s 
computer lab. 



FIGURE 2 

LOTUS DEVELOPMENT CORPORATION 




11^; AWEGlATION OF GENEROUS 
gUPlPOPt THIS COMPUTER LAB IS 
?! " ^ NAMED IN HONOR OF 
^46tuS DEVELOPMENT CORK 



The Lotus donation, however, was not without its 
detractors. There were a number of faculty members 
who resented having not having a choice about the 
applications to be used in particular courses. Others 
questioned whether any company or donation should be 
allowed to dictate course/curriculum content. Most 
detractors, however, were grateful for the COB A 
resources purchased by the $1 million in matching 
lottery money that flowed into the college as a result of 
the donation. 

The IBM AS/400 Education Partnership Program 

Another program that has had an impact on the IS 
program at Georgia Southern is the IBM AS/400 
Education Partnership Program. This program was 
available to technical schools during the early 1 990s, but 
in 1996, it was expanded to include four-year colleges 
and universities in order to increase the number of 
entry-level system/business analysts with AS/400 
experience for IBM Business Partners. The program 
included the donation of an AS/400 from IBM. The 
hardware donation was worth $ 1 00,000 and qualified for 
matching lottery funds. The machine was installed and 
maintained by an IBM Business Partner agreeing to 
serve as the university sponsor. Faculty training (a 
mandatory part of the program) was provided by IBM. 



In return for the donation, the university agreed to 
integrate the machine into its curriculum. In our case, 
this meant creating and implementing an AS/400 
Applications course. Student benefits from the program 
include the opportunity to acquire some niche skills that 
differentiate them in the entry-level job market 

MK Logistics 

In another example of collaboration, the MK Group of 
computer associates in Atlanta (now part of InterBiz) 
provided its logistics software to the COB A in order to 
enhance the opportunities for logistics majors to gain 
hands on experience with logistics, warehouse 
management, and supply chain management software. 
InterBiz provides training for faculty on how to 
incorporate the software into their courses. In return, 
InterBiz requires documented evidence of its use in 
business courses. 

SAP Education Partnership Program 

The donation that has had the farthest-reaching impact 
on COB A ’s curriculum is the SAP partnership. This was 
initiated soon after the MK Group’s logistics software 
donation, but is yet to be fully implemented. More than 
a half-dozen faculty members have had SAP training. 
Most business disciplines within COBA have been 
represented in the training including Accounting, 
Finance, Operations, Logistics, and Information 
Systems. While there is no cost for the training itself 
COBA has had to bear considerable travel and 
accommodation expenses associated with the training. 
And, although the software is donated, an appropriate 
hardware platform had to be purchases. Other associated 
costs have included the salary and benefits of an SAP 
administrator. As a result of the startup and ongoing 
costs, several administrators have come to call the SAP 
initiative “the gift which keeps on taking.” 

The SAP partnership agreement calls for the software to 
be used in multiple courses by students in several 
majors. Thus far, business students are working on SAP 
tutorials in Introduction to Business course, in some 
accounting courses, and in an advanced operations 
management course. Thanks to COBA’s ability to hire 
a faculty member with extensive SAP training and 
experience, beginning next fall, students will also be 
able to take an Introduction to SAP course and over the 
next academic year, an ERP course will also feature 
hands on experience with SAP as its centerpiece. An 
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SAP Programming course is a planned addition to the IS 
curriculum and SAP’s HR module will be featured in 
new Human Resource Information Systems course. 

Students in a variety of majors are excited about the 
opportunity to gain hands on experience with SAP. They 
tend to see SAP experience as something that will 
differentiate them from other undergraduate business 
students competing for entry-level jobs. They are putting 
pressure on COBA administrators to roll out the SAP 
courses as quickly as possible. The creation of such 
grassroots demand among students is probably exactly 
what SAP hoped would happen. Such ongoing demand 
translates into continued marketing for their products. 

NCR POS Outsourcing Agreement 

The most recent donation that is having an impact on 
Georgia Southern’s IS program is the result of an 
outsourcing agreement between the university and NCR. 
NCR has outsourced the ongoing development of one of 
its major POS products to Georgia Southern. This 
startup has resulted in a faculty member being relieved 
of all teaching duties until a full-time project manger is 
hired to oversee the project and the students that will be 
working on it. The agreement between NCR and the 
university specifies the creation of several new courses, 
both graduate and undergraduate in addition to graduate 
and undergraduate internships to ensure on^ing work 
on the upgrade. Thus far, one new course has been 
created as part of this new (Fall 2001) agreement. More 
are planned. 

This agreement has the potential to result in $50,000 to 
$150,000 in revenue from NCR for COBA and the 
Department of Information Systems and Logistics. It 
also provides students with the opportunity to be directly 
involved in the ongoing development of a widely used 
product — an experience that students at most other 
universities never get. The potential to gain a 
differentiating skill set in the market has inspired many 
of our best students to apply for the internships. 

Other New Initiative? 

COBA’s Department of Information Systems and 
Logistics is currently considering other potential 
donations that are likely to have curriculum-altering 
strings attached. COB A ’ s Acting Director for the Center 
for Logistics and Intermodal Transportation is confident 
that he can secure the donation of Manuguistics, 



arguably the leading logistics and supply chain 
management software in use today. The donation would 
be valued at approximately $1 million dollars and is 
likely to be followed by scholarship donations from 
major Manuguistics users. Providing logistics and IS 
students with hands-on experience with the software 
would instantly differentiate them in the marketplace 
and provide national visibility for GSU’s logistics 
program. In exchange for the software, COBA would be 
required to integrate Manuguistics in multiple logistics 
courses and create a Manuguistics programming/ 
applications course for IS majors. 

A final initiative under consideration is the potential 
donation of a customer relationship management (CRM) 
product and its integration in marketing and IS courses. 
Once again, students would be able to gain a skills set 
that would differentiate them in the job market 

MUTUALLY BENEFICIAL COLLABORATION? 

The generosity of vendors and other corporations is not 
solely based on their desire to address the ongoing 
technology currency challenges faced by universities. 
Typically, they look at their donations as a form of 
strategic philanthropy that will result in a long-term 
return on investment , a gesture that is likely to increase 
their chances of hiring the best and brightest students 
graduating from the technology programs they have 
chosen to donate to, and an opportunity to create loyal 
consumers of their products. 

The discussion of corporate partnership with secondary 
schools and universities must also include an analysis of 
the intangible benefits and consequences of such 
collaborations. Different views toward corporate 
involvement with secondary schools has been 
documented On one side. Dee McNosky, a special 
education teacher in Texas indicated her support of 
sponsorship by referring to corporations as another 
“stakeholder in the future of public education” 
(McNosky & Gump, 1998, p. 59). She believes that 
everyone both individually and collectively, should be 
involved in public education including corporations. On 
the opposing side, Judy Gump, a teacher in Wisconsin 
commented that 

Children have the right to a quality education, 
one that allows them to make choices about 
their futures. When corporations worm their 
way into the schools, they make the choices. 
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they set the agenda (McNosky & Gump, 1998, 
p. 59). 

Both have very valid points. A major concern is how 
much influence corporations have over secondary 
schools and universities as the result of a donation or 
partnership agreement. Are colleges and universities 
allowing businesses to become owners of education and 
not Just participants in the education process? Is this not 
what is happening when the curriculum is adapted to 
meet the conditions of a donation or the requirements 
partnership agreement with a vendor? 

At the university level, students are not being trained for 
“low-skilled, low-pay jobs” as Henry Giroux (2000) 
would suggest. Data from the U.S. Department of 
Commerce suggests just the opposite, upon graduation, 
they will be entering some of the most highly skilledand 
highly paid occupations in the U.S. workforce. In many 
cases, students have a higher income than their 
professors within five years after graduation. This is 
especially true if they had the opportunity to develop a 
differentiating skill set prior to entering the job market. 
This suggests that adapting curricula to provide students 
with hands on experience with widely used vendor 
products may, in fact, benefit students. 

FIGURE 3 

CHICK-FIL-A CLASSROOM 




Some individuals in higher education maintain that we 
must not never become complacent about corporate 
collaboration. As noted by one group, corporate dollars 
“come with too many strings attached” (Mangan, 1999, 
pg. A 14) and are a threat to academic freedom. In 
addition, there is concern that corporate support for 
research can undermine objectivity. Some professors 
fear that they may be political consequences if you 
conduct research in an objective fashion and come up 



with results that are not favorable to the sponsoring 
industry. This could threaten the university’s ability to 
attract corporate funding in the future and could cause 
be a loss of a tenured professor’s job if he/she was 
discovered to be fudging results in order to make the 
sponsor look good. In order to help protect a 
researcher’s academic freedom, the American 
Association of University Professors, AAUP, approved 
a statement that reads as fol lows 

The freedom to pursue research and the 
correlative right to transmit the fruits of inquiry 
to the wider community- without limitations 
from corporate or political interests and without 
prior restraint or fear of subsequent 
punishment-are essential to the advancement of 
knowledge ( Mangan, 1999, pg. A 14). 

This is a bold statement, but it may not carry much 
weight in application. Researchers cannot control the 
purse strings of corporations and if they want to 
withdraw their money they can do so. In addition 
because most partnerships include formal and informal 
associations between corporate executives and academic 
administrators, the opportunity exists for executives to 
unduly persuade administrators to restructure the 
distribution of funding to professors. 

SUMMARY 

The foregoing discussion indicates that there are both 
advantages and disadvantages for universities involved 
with the strategic philanthropy of corporations. Funding 
and other economic realities facing universities increase 
the attractiveness of vendor donations and education 
programs, even when there are obvious strings attached. 
Both tangible and intangible benefits and costs should 
be weighed when considering \\diether to accept a 
donation or enter a partnership program. 

Impacts on Curriculum 

Corporate/vendor donations and education partnaship 
programs confront educators with the perennial question 
of training vs. education. If we accept the donation and 
alter course content in order to integrate the 
software/hardware in the curriculum, are we moving 
away from education and toward training? Otha 
important questions include 

• Are donated software modules/applications the “best” 
ones for illustrating the enduring concepts needed for 
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lifelong learning? Are the technologies truly best of 
breed? 

• Where does the donated technology stand in terms of 
product obsolescence? Is this new or old technology? 

• Will student exposure/use truly translate into 
better-prepared students? 

• Is it possible to deliver an equivalent or superior 
curriculum and educational experience without this 
technology? 

• Are we caving into vendor pressure and/or 
administrative pressure just to save money? 

• Have we identified all potential costs and/or benefits? 
What hidden costs may jump up to bite us? What 
additional benefits might materialize down the road? 

• Are we ready, willing, and able to live up to our 
contractual agreements? Are we willing to make 
adjustments to the curriculum called for by the 
donation or partnership agreements. 

• Are we ready, willing, and able to take full advantage 
of the donation or partnership program? 

• Is this really in the best interests of our students? Of 
the program? Of the university? 

• Is the cost of doing nothing greater than that 
associated with saying no to this donation or 
partnership agreement? 

Developing honest answers to these questions may be 
one of the best starting points for assessing the impacts 
of strategic philanthropy on your university’s IS/IT 
program. 
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ABSTRACT 

Almost all undergraduate business programs offer a ''capstone” course in business policy or strategy. In addition, 
may of these programs offer ''capstone ” courses in the various majors or disciplines, usually focusing on discipline- 
level strategy, such as an MIS strategy course. Typically, some of the content in the organizational-level strategy and 
discipline-level strategy courses is duplicated. Where exactly dowe draw the boundaries? What are the implications 
for students? How can the duplication and boundary overflows between these different types of strategy be reduced 
or eliminated? We make three recommendations that we find helpful in our teaching of these two strategy courses. 



For most undergraduate business programs, the Business 
Policy or Strategy (BPS) class is the capstone course, 
incorporating all of the foundation course knowledge 
developed during all business students’ undergraduate 
program course work. The knowledge incorporated in 
the BPS course includes material from finance, 
marketing, accounting, management information 
systems, human resources, organizational theory, 
organizational behaviour, and operations management. 
The BPS course prepares the student for the real world 
and for making sense of the business world by forcing 
them to integrate knowledge gained in a variety of 
courses taken over two or more years. Most often, the 
business policy or strategy course is taken during the 
last or next to last semester the student is enrolled prior 
to graduation. 

For many management information systems (MIS) 
undergraduate programs, the MIS Strategy course is the 
capstone course for the major and incorporates all of the 
knowledge learned earlier in the MIS major along with, 
typically, much of the knowledge listed above for the 
Business Policy or Strategy (BPS) course. The MIS 
capstone course has the same purpose as the BPS 



course, but with a narrower focus, highlighting the 
world of the IS professional. Many times, the MIS 
course may be taught during the same semester as the 
BPS course, but it is frequently taught prior to the BPS 
course. 

Both the BPS and MIS capstone courses contain much 
of the same content and examples, use the same models, 
and occasionally, even the same cases. Regardless of 
which semester an MIS major takes the MIS capstone 
course, there will be a considerable amount of overlap 
between the two courses. This results in a variety of 
potential problems and duplicated efforts. The purpose 
of this paper is to delineate those problems and areas of 
duplication and to make recommendations on how to 
resolve the problems and decrease the duplicatedefforts 
without losing any of the substance or rigor of either of 
these two critical-thinking courses. This is a delicate 
task undertaken by colleagues with the overall goal of 
enhancing the students’ educational experience and 
knowledge. Indeed the boundaries between the two 
courses may be seen as fluctuating. Determining 
boundaries is a sensitive task, but a necessary one. 
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Our purpose in this paper is to uncover areas of potential 
overflow between class boundaries and to discuss ways 
in which these boundary overflows can be accom- 
modated to best suit both students and feculty. We begin 
with an outline of some of the conceptual issues 
involved in strategy. We continue with an examination 
of the way in which these conceptual issues are applied 
in MIS and in BPS. We then offer an example of how 
the same model can be taught from each perspective. 
Finally, we offer some general recommendations on 
how to deal with duplication and boundary overflow in 
these two important subject areas. 

POTENTIAL PROBLEMS AND 

DUPLICATION OF MATERIAL 

The concepts “strategy” and “strategic” are common to 
both MIS and BPS capstone courses. Thus, one of the 
first tasks is to ensure that students understand these 
terms and their implications. One of the first difficulties 
students have is discriminating between strategies and 
tactics. Strategy is a plan designed to achieve a long- 
term aim, and strategic implies actions that are part of 
along-term plan or aim to achieve a specific purpose. 
Tactics, on the other hand, are actions that aim to 
achieve a specific end. While the definitions may seem 
ambiguous to some students, it is important to highlight 
the difference. Strategies are planned in order to 
achieve an end beyond the immediate result. Thus, in the 
MIS area, moving to client/server (tactic) may result in 
reduced profit over a particular period, but it should also 
result in much more flexibility and scalability as well as 
reduced costs in the long term (strategy). 

Once the students have understood that “strategic” 
requires them to focus on the long-term, they can move 
on to an examination of what strategic means in the 
context of their course. In the BPS course, it is important 
to discuss business strategy and corporate strategy. 
Business strategy answers the question: How do we do 
business? Corporate strategy, on the other hand, answers 
the question: What business (or businesses) are we in? 
Each level of strategy requires analysis and visioning, 
followed by implementation plans. When we discuss 
implementation, the role of functional strategies will 
inevitably arise. Functional strategies are multi-purpose; 
they are crafted to meet area objectives, but also to 
support the organizational strategy. This is where the 
BPS instructor can make important links between MIS 
and BPS. Functional strategies are meant to be 
supportive of, rather than subordinate to, BPS strategies. 



Thus, it is important to frame discussion around the 
integration of business-level, corporate-level, and 
functional-level strategies, and to show how the 
successful integration of these strategies is essential for 
the organization to achieve sustainable competitive 
advantage. 

In the BPS course at our school, we spend two classes 
examining ways in which functional strategies lead to 
distinctive competencies and competitive advantage. 
While some areas such as production and purchasing are 
easier to illustrate, MIS can sometimes be more 
difficult. Because it is often not directly a revenue- 
producing area, MIS can often be seen as simply 
supporting secondary activities, and not crucial to 
competitive advantage. It is important for both BPS and 
MIS instructors to demonstrate the fallacy of this 
argument through concrete examples showing how MIS 
strategy can be both supportive of primary activities and 
crucial to the firm’s competitive position. 

Similarly, in the MIS Strategy class at our school, we 
spend two classes at the beginning of the semester 
discussing the definitions of strategy, business strategy, 
MIS strategy (and other functional strategies), and the 
role of the functional strategies in supporting, enabling, 
and informing the organizational strategy. We frame the 
differences in the two types of strategy and discuss the 
need for alignment of MIS strategy with the organiza- 
tional strategy as well as the need for deliberation and 
purpose in aligning the two. The next few classes are 
spent discussing how specifically does MIS enable 
business/corporate strategy, from various analytic 
techniques, such as Porter’s Five Forces model, to 
implementation models. Many of the models and 
techniques we use are the same as those used in the 
Business Policy class, but the focus is much narrower, 
looking only at the MIS strategy and how it can align 
with and enable the organizational strategy. 

Important models like Porter’s Five Forces model can be 
powerful tools for understanding critical concepts in 
MIS and BPS. Students will therefore likely be exposed 
to a model like Porter’s several times during their 
undergraduate career. There is the danger that, after the 
first exposure, they will pay less attention to the model 
and will thus lose sight of its applicability in a different 
context. BPS students will often state that they have 
already seen Porter in Marketing and MIS, at the very 
least, and will thus be tempted to gloss over the 
explanation of Porter’s model in a BPS context The 
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solution is not to decide to cut out use of the model in 
another course, but rather to discuss the model from the 
perspective of the course in vdiich it is taught. 

Thus, for example, an examination of Porter’s Five 
Forces model in the MIS capstone course typically will 
involve a detailed analysis of one of the forces and how 
information systems and technology (1ST) can alter that 
competitive force and its impact on the organization. For 
example, American Airlines SABRE reservation system 
raised considerable barriers to entry for potential 
competitors. In this example, the SABRE system 
supported the organizational strategy of preventing new 
competitors from entering their market. Students are 
frequently then asked to look at the other forces in teams 
and report back at the end of the class on how MIS 
strategy can affect those four forces and the 
organizational strategy. 

In the BPS course, on the other hand, it is important to 
develop the use of Porter’s Five Forces as a tool for the 
evaluation of the firm's external environment. This 
discussion should keep in mind the opportunities that 
arise from the environment. Duringthe internal analysis, 
the capabilities of functional areas such as Misbecome 
critical in the development of the firm’s strategy. A 
thorough environmental analysis using Porter’s model, 
done from both a firm and MIS perspective, will make 
the development of a successful strategy that uses core 
competencies effectively that much more likely. 

RECOMMENDATIONS 

There are many different “types” of strategy, or 
disciplines in which strategy is a primary factor. 
Universities offer courses in financial strategy, 
marketing strategy, MIS strategy, HR strategy, and other 
types of strategy. Each of these courses is usually the 
capstone course for that discipline’s major. Each of 
these types of strategy is a support tool or enablerof the 
overall organization’s strategy. Yet the difference in the 
discipline-level strategy and organizational level 
strategy can become obscure when teaching the 
discipline-level strategy classes that are focused on the 
narrower, enabling strategies. Anecdotally, we have seen 
students in business policy classes say that MIS strategy 
is the organization’s strategy, and students in MIS 
strategy classes have said that only the organizational 
strategy matters — neither of which represents the 
thinking or teaching of the authors. Given the discussion 
above, we offer the following recommendations to help 
integrate and differentiate these concepts. 



Recommendation 1 

Teachers in the discipline-level strategy courses should 
clearly and repeatedly emphasize the difference in 
organizational strategy, discipline-level strategies, and 
their own discipline’s strategy. For example, in teaching 
MIS strategy, instructors should point out the 
differences in organizational strategy, the various 
discipline-level strategies and the ways they enable 
organizational strategy, and MIS strategy and its 
specific focus in enabling organizational strategy. By 
repeating these differences and by being clear and 
focused on these differences, students should better 
learn the differences and how these areas overlap. 
Students should better understand that the boundaries 
fluctuate between which discipline’s strategies enable 
which organizational strategies and which discipline’s 
strategies are more important to a particular organiza- 
tional strategy than others. 

Many times, in both discipline-level strategy classes and 
organizational strategy classes, students hear the same 
models being explained. Frequently, even the same 
cases are used. For example, Wal-Mart is a frequently 
used case in both MIS and organizational strategy 
classes. Yet the approaches of each case discussion 
shouldbe different.Porter’scompetitive forces model is 
almost always used in both of these classes. Again, the 
approaches should be different, with MIS students 
focusing, for example, on how information technology 
can affect the five forces and the way the forces affect 
competition while organizational strategy students focus 
on how information tedinology can help the firm sustain 
competitive advantage in a given, sometimes changing, 
environment. 

Recommendation 2 

T eachers should clearly delineate the different emphases 
of each model and case to continue efforts to show the 
boundaries and differences between organizational 
strategy and discipline-level strategies. If the proper 
emphasis is placed on a particular view of a model or 
case, students will learn that there are many views 
possible for these models and cases, and that the 
distinctions and differences are significant. 

The organizational strategy course should typically be 
taken in the last semester of the student’s undergraduate 
career. Frequently, students have taken enough courses 
in their major to take their discipline-level strategy 
course during an earlier semester. For students who take 
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the discipline-level strategy class earlier, the discipline- 
level strategy class must discuss organizational strategy, 
including explaining various models, such as Porter’s 
competitive forces model, in order to proceed with 
discussions of discipline-level strategy. 

Because most of the models and techniques that are used 
in both courses are frequently introduced in other 
courses, such as Introduction to MIS, Introduction to 
Marketing, or Organizatiai Theory, it may be possible 
in the discipline-level strategy course to abbreviate the 
discussion of some of the models and techniques and 
focus only on their importance to the issues involved in 
the discipline-level strategy. The BPS class, on the other 
hand, would more likely discuss in greater detail these 
models and techniques and their importance to 
organizational strategy, taking into account their 
importance to the discipline-level strategies as well. 

Recommendation 3 

Instructors should, at the same time, discuss and 
evaluate areas of crossover or boundary overflow 
between strategies and strategic tools. It is crucial for 
students to understand that strategies must work in 
tandem in order for the firm to receive the most benefit. 
The first step in learning should be to clearly 
demonstrate how strategies are developed and applied in 
separate areas, as per recommendations one and two 
above. However, in order to move the students to a 
higher level of understanding, it is important to follow 
this with a discussion of how functional-level and 
business-level strategies work together. 

Discussions focused on alignment and relationships 
between the functional and business strategies can help 
in this regard. In addition, discussion in the discipline- 
level strategy class of the importance of the other 
discipline-level strategies is also important. An example 
or two where the other discipline’s strategy supersedes 
the strategy being taught in the class usually clarifies 
this for the students. 

Recommendation 4 

Finally, it is important for instructors to work together, 
across disciplinary boundaries, to ensure that there is 
less direct overlap, more integration, and more 
coherence in the way strategy is taught in their school. 



We have made a case for teaching strategy at both a 
discipline and a firm level. We have also made a case 
for emphasizing the importance of integrating these two 
levels of strategy in order to help the firm gain 
sustainable competitive advantage. Our colleagues in 
Organization Theory would urge the firm to develop 
boundary-spanning activities and create communication 
pathways in order to ensure this integration. We 
recommend that faculty members do the same. It is 
important to work with colleagues in other areas to 
discuss and revise how strategy is taught in different 
courses in order for students to derive the maximum 
benefit from their undergraduate business education. 

For example, the sharing of syllabi and other course 
materials, such as a list of cases to be used in each class, 
can pave the way for a discussion of how the types of 
strategy are different and should be taught differently. 
Not everything that has strategy in the title is the 
deciding factor in how organizations operate and thrive. 
It is the integration of the strategies, working together, 
that accomplish this. In the same way, instructors 
working together can ensure that the concepts of what is 
strategic, what provides competitive advantage, and how 
these strategies can work together are understood by 
their students. 

CONCLUSION 

We have shown the similarities in discipline-level and 
organizational strategy classes and the duplication and 
conflicts that can arise when students take both courses. 
A cursory review of comparative syllabi at other 
universities shows that our findings of duplication are 
accurate. We recommend that instructors emphasize the 
differences in the two types of strategy and where the 
discipline-level strategy courses fit with the organiza- 
tional level course. We recommend that instructors 
emphasize the enabling aspects of discipline-level 
strategies rather than making them appear to be the 
overriding organizational strategy. We recommend the 
inclusion of the concepts “strategy” and “strategic” from 
the start of the undergraduate business degree, so that 
students learn the different and nuanced ways they can 
be applied. Finally, we recommend that instructors of 
these two types of strategy classes work together to 
ensure that they emphasize different views of models 
and cases. 
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ABSTRACT 

This paper addresses the current state of B2B electronic commerce and identifies the major reasons for slower 
adoption of B2B electronic commerce than was beingforecast by the media, consulting firms and market research 
firms. Understanding the reasons for slower than expected adoption and growth rates provides business leaders with 
insights as they move forward in designing strategies for future business models and defining partnerships for 
electronic B2B commerce. This research represents a synthesis of current thought from the literature and the 
experiences of one major player in the B2B e-marketplace space. 



INTRODUCTION 

Much has been written about the many facets of B2B 
electronic commerce. More recently the focus has 
shifted to B2B exchanges and e-marketplaces that 
include consortia, independent marketplaces, and private 
marketplaces. As these models have evolved over time 
much has been published in the trade literature 
forecasting the success and rapid adoption by the 
various players in the business community. The most 
current trade literature has taken a turn toward 
addressing the issues of what went wrong and the 
reasons for slower than expected adoption rates and in 
some cases failure. The literature base for the research 
reported in this study comes primarily from the business 
community trade literature including white papers, 
industry reports, and company research. Very little has 
been reported in the academic literature with a focus on 
the issues of failure and the reasons for slow B2B e- 
commerce adoption rates. Although some related 
articles have been published (e.g., Kaplan and Sawhney, 
2000; Han andNoh, 2000; Hope, Hermanek, Schlemmer 
and Huff, 2001; Essig and Arnold, 2001; Wise and 
Morrison, 2000) we did not find any academic work 



addressing the specific tq)ic of this paper. This is likely 
due to the short time frame in which it has even been 
possible to study these issues. Given the time required 
to publish in most academic venues any comprehensive 
study of B2B e-commo'ce failure trends and growth 
rates would likely not yet be available to the public 
audience. 

This work provides a foundation for more extensive 
examination of issues, trends, and management 
strategies for B2B e-commerce in the future. 

OVERVIEW OF REASONS 
FOR SLOW ADOPTION 

The Internet is redefining how businesses communicate 
and trade. Business processes and entire supply chains 
have been restructured and new business models have 
emerged. Although the Internet is dramatically affecting 
the way we do business, the rate of growth and adoption 
has been much slower than expected. 

We have identified many reasons for this phenomena. 
Two primary reasons are 1 ) marketplaces and exchanges 
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have not flourished and 2). the failure of companies to 
recognize the essential role of content. We will focus 
on these two major reasons for slow adoption in this 
study. 

During the prosperous years represented by the e- 
commerce boom when the stock market soared and the 
hype was promoted by the press, many were persuaded 
that the pace of change reflected revolutionary 
transformation. In reality, however, it has taken much 
longer than anticipated for companies to transition to 
transacting B2B electronic commerce over the internet. 
B2B exchange transaction volumes, for example, are 
quite a bit lower than the original analysts forecasts. 

It was a misconception to believe that B2B e-commerce 
and marketplaces were practically a guaranteed success. 
Although many companies will abandon traditional 
ways of doing business for the potential benefits of the 
new, the transitions will take place to varying degrees 
and within varying time frames depending on company 
characteristics and industry variables. It can be 
concluded that for most industries full scale B2B e- 
commerce has not yet materialized at the rate and degree 
predicted. 

As an example, the rate of adoption has been measured 
with B2B media. At the beginning of the Internet boom, 
many assumed that the Internet would rapidly replace 
traditional media for the many “obvious” advantages. 
However, one recent study conducted by Yankelovich 
and Harris Interactive (2001) in the U.S. showed that 
professional trade magazines and journals still rank 
ahead of web sites as executives’ most important source 
of information. A major conclusion from the study was 
that executives seek information from various sources 
and feel that a combination of information sources is the 
best strategy for staying on top of developments in their 
field. 

In hindsight, the dangers of moving ahead based on the 
response of early adopters should have been evaluated 
more carefully and with greater scrutiny. Although 
there were a large number of good ideas, they were 
chasing a comparatively small potential market. As 
exemplified by the work of Geoffery Moore (1999), 
most companies are not early adopters of cutting edge 
technology. According to Moore, there is the danger of 
not being able to “cross the chasm” from early adopters 
to the point where there is a majority of adopters. 
Crossing this gap is the place where many companies 
fail because they miscalculate the time and effort 



required to move from the early adopter phase to that 
where a majority of adopters accept the technology. 
B2B e-marketplaces, for example, have not progressed 
as rapidly as predicted due to this slower pace of 
adoption from both the buyer and supplier side. 

The process has been more difficult than most 
companies planned for. Business, however, has been 
forever changed by the digital transformation of 
processes and restructuring of supply chains. Progress 
will continue as companies extend advancements made 
within their boundaries to process improvements 
external to the organization. Hammer (2001) advocates 
that this will be the focus of company agendas during 
the next decade. The question at this point is not 
whether B2B e-commerce will flourish but how fast the 
rate of adoption will be. 

E-MARKETPLACES HAVE NOT FLOURISHED 

B2B marketplaces were prominent in the press and at 
industry conferences for most of 2000 and 2001. 
Countless articles, books, and papers have addressed the 
many facets of B2B e-commerce. Many start-ups 
entered the space seeking first mover advantage while 
others sought competitive advantage through consortia 
and exchanges. The hype created by the media and 
market forces overshadowed reality and practicality. 
(Laseter, Long, and Capers, 2001). 

The number of marketplaces rose to approximately 2000 
globally as companies rushed to take part in the B2B 
marketplace gold rush (Laseter, Long and Capers, 
2001). These B2B marketplaces promised productivity 
gains, cost reductions, revenue generation, and increased 
efficiency in matching buyers and suppliers (AT 
Kearney, 2001). 

Growth in marketplaces began to decline in 2001 and 
has continued on a downward spiral since then. Many 
of these e-marketplaces could not survive and were 
forced to close. BIB marketplaces and exchanges were 
among the very hardest hit when the dotcom stock 
market bubble burst. After running up to unbelievable 
levels, the typical marketplace stock is down by 90 to 99 
percent. 

Reasons for the Failures 

In addition to cultural and technical obstacles, there are 
a multitude of economic, business and management 
issues which led to the downfall of BIB marketplaces. 
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• Greed led to the collapse of many e-marketplaces. 
There was an amazing gold rush mentality that went 
beyond reason. Business models that didn’t make 
sense got funded to try to capitalize on the huge stock 
market valuations. Companies built something online, 
but most didn’t meet the needs of an industry (Lay, 
2000 ). 

• Business models of most B2B exchanges were not 
well developed. They did not take the time to study 
customer expectations or develop a path to 
profitability. (Wise and Morrison, 2000). 

• Procurement may not be the “killer app” for net 
markets that was assumed. To many, e-canmerce is 
more about collaboration and streamlining supply 
chains. (Murphy and Baling, 2001). 

• There was a lack of standards to promote integration 
across company sites, distributor sites, and industry 
sponsored exchanges with back end systems (IDC, 
2001 ). 

• Companies underestimated the value of catalog 
content in the success of e-procurement projects (Pod, 
2001 ). 

• Very few e-marketplaces have attained the critical 
mass of participants required to deliver the anticipated 
benefits. Too many buyers and suppliers are still 
waiting to see how this concept will develop. (BCG 
Research, 2000). 

• Although marketplaces needed a content and catalog 
management strategy to be successful, few got it right. 
And still today, very few marketplaces possess the 
expertise or resources for acquiring, transforming 
categorizing, managing and distributing catalog 
content (Knickle, 2001). 

• Excessive management hype created inflated 
expectations as to what marketplaces would do and 
how quickly they would be up and running. When 
reality didn’t keep pace with the promises, many 
became disillusioned. 

• Companies were surprised by the complexity of 
running an exchange. They struggled to develop the 
infrastructure, service offerings, and the business 
expertise required. Running a marketplace, selling it 
to buyers and suppliers, content mapping, and 
transaction enablement is not a core competency for 



many businesses other than those devoted to running 
marketplaces. (Cap Gemini and Eamst & Young, 
2000 ). 

• Companies decided to turn inward and focus on 
private exchanges that linked them with key partners 
and suppliers. The objectives were not to change 
processes, but make them more efficient (AMR 
Research, 2001). 

• Suppliers were unwilling or unable to participate for 
various reasons. In particular, suppliers did not rush to 
join the many buyer-centric marketplaces where the 
primary application was reverse auctions aimed at 
squeezing supplier margins (IDC, 2001). 

• A lot of e-marketplaces had an excessive focus on the 
technology in the mistaken belief that “if we build it 
they will come”(Fingar and Aronica, 2001). 

• Consortium exchanges were stymied by huge 
integration barriers, runaway costs, technological 
complexity, administrative bickering, rivalry among 
competitors, and a souring market for B2B exchanges 
(IDC, 2001). 

Survival of the Fittest 

Despite the problems, optimism regarding the future of 
exchanges is strong. Nearly half of all companies polled 
in a Giga Group study (2001) expect exchanges to 
improve profitability and create standard business 
practices. Another study by Gartner (2001) states, “By 
2005, publicandprivatee-marketplaces will become the 
dominant B2B trade mechanisms for thousands of 
companies.” The overall concept of marketplaces is 
sound. Marketplaces are the next wave of business 
transformation. The challenge is for e-maikeplaces to 
create the critical mass and liquidity necessary for a 
viable business (Boston Consulting Group, 2001). 

This will not be easy for most players in this space. 
Boston Consulting Group (2001) predicts that the US 
B2B marketplace will be reduced to a handful of e- 
marketplace giants serving a given industry. Many 
niche players will emerge to address specific needs of 
industry segments. It will be survival of the fittest with 
significant opportunities for those who survive. 

A prerequisite of survival and success is to have a 
business model that is appropriate to the industry — and 
then to execute effectively. However, decisions as to 
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what is the right business model vary tremendously by 
industry — and by where the marketplace fits in the value 
chain. Clearly there is no magic formula for selecting 
the right business model since most surviving 
marketplaces have evolved their models several times, 
often quite dramatically. 

For consortium exchanges two factOTS will ultimately 
drive their success or failure. They will need to create 
services that become an industry standard. In addition, 
they need founders that remain committed to ensure the 
financing and usage fees necessary to keep the exchange 
afloat while it builds capabilities. For independent 
marketplaces it’s essential to focus on the bottom line to 
conserve cash. Launching bleeding edge new services is 
no longer affordable. Independents that are clearly 
focused on a niche market and/or application will have 
the best chance of success — if they have built a 
sustainable customer base and deliver a clearly visible 
value proposition (Laseter, Long and Capers, 2001). 

CONTENT WAS 

NEGLECTED AND UNDERVALUED 

If one looks back at all the expert views on what it was 
going to take for B2B commerce and marketplaces to 
flourish, what stands out is that many grossly 
underestimated the value of content and the difficulty of 
creating and maintaining it. 

The early focus on transactions from the sell-side, and 
the buy side was a primary reason for the lack of focus 
on content and catalog issues and thus jeopardized all e- 
commerce initiatives. Most procurement applications 
and marketplaces have failed to deliver the benefits 
expected primarily because of the difficulty of acquiring 
and managing content. In fact, the lack of suppliers with 
digital catalogs may be the biggest factor limiting the 
growth and widespread adoption of B2B e-commerce. 

Why Content Was Neglected 

How could so many smart people either not appreciate 
the importance of content, or not allocate the necessary 
resources to content? The reasons for this state of 
content, and particularly supplier catalogs, are many. 

Assuming traditional hub and spoke behavior was a key 
reason. Just like years ago when buyers demanded that 
key suppliers move to EDI or lose the business, many 
expected that suppliers would be forced by buyers to get 
their catalogs digital. The lesson this time is that if you 



build an exchange or marketplace, suppliers will not 
come — without convincing and assistance. 

Assuming that digitizing catalogs was a no-brainer 
today for suppliers was another key reason. However the 
software, tools, standards and services are still immature 
and evolving. Digitizing catalogs is a major change for 
suppliers that impacts many systems and business 
processes. While marketplace operators and e- 
procurement software vendors paid close attention to the 
giant distributors whose catalogs were the foundation 
for MRO procurement, no one was out getting face-to- 
face with the thousands of other suppliers. The reality is 
that most suppliers are not ready to develop and manage 
digital catalogs on their own — and that they require 
education, handholding, software, service and support. 

Many who took content for granted were shocked to 
realize that content acquisition and maintenance is hard 
work and that there are no shortcuts. In addition, there 
was a shortage of firms who were even able to do it. 

Lastly, there was a lack of clarity about who would pay 
for content acquisition and maintenance and even today, 
it ’ s still unclear with an evolving combination of buyers, 
suppliers and marketplaces. E-procurement software 
vendors and marketplaces lacked business models that 
addressed the high cost of content. They seemed to hope 
that somehow, someone else would do it so that they 
could focus on large buying organizations and on the 
more glamorous and lucrative areas of transactions, 
collaboration and integration. 

Content Reduces Interaction Costs and Creates 
Value 

Today, success for many businesses depends on 
achieving advantages in the manufacturing process. 
This has driven firms to concentrate on core 
competencies and core products, which is behind the 
huge shift to outsourcing, and private label and offdiore 
production. These trends have, in turn created high 
interaction costs (Essig and Arnold, 2001). 

According to a McKinsey report (Bryan and Fraser, 
2001), “interaction costs — expenses generated by the 
need to coordinate work among different parties — now 
represent as much as 51% of labor activity in the United 
States.” The fundamental value of electronic and 
structured content is in it’s ability, along with 
technology to reduce the huge interaction costs that 
businesses suffer from today. However, in addition to 



90 



Proceedings of the 17^ Annual Conference of the International Academy for Information Management 



cost reduction digital content also expands the 
possibilities that purchasing and/or selling will create 
value for the firm. Better information and content can 
create significant competitive advantages for buyers and 
sellers. 

Content Fuels Commerce and Exchanges 

As emphasized in a recent report promoting the value of 
B2B content networks (Kinecta, 2001), content will 
drive every interaction, transaction and activity across 
the supply chain with customers, suppliers and partners. 
Product catalog information in particular, is the fuel for 
commerce. Without the electronic catalog there is no e- 
commerce. Content is also the fuel for exchanges. 
According to AMR Research (Knickle, 2001), “content 
is the heart of all marketplaces-private, consortium, and 
public.” That’s because incomplete or inaccurate 
content, or an ineffective search engine will 
dramatically reduce the value of a marketplace for all 
participants. 

Since inefficiencies in purchasing are largely 
information driven, the availability of greatly enhanced 
information and content rich marketplaces gives buyers 
the ability to be much more efficient and effective 
(Essig and Arnold, 2001). According to a study by A.T. 
Kearney and CAPS (2001), sourcing represents 73% of 
the total procurement savings opportunity. Aberdeen 
Research (August 2000) had a similar finding, saying 
that “the initial sourcing process, from supplier search 
and identification through contract negotiation and final 
purchase, has the greatest impact on the cost, quality, 
and delivery of a product or service.” 

Since content fuels business activities, the value of a 
business network is highly correlated with the flow of 
content through the system to participants. The ability 
to get the right content to the right people in a defined 
format and time frame will be critical to the future 
success of companies in the B2B marketplace space 
(Kinecta, 2001). 

Barriers to Content 

High quality content is in much higher demand than the 
available supply. However the barriers are significant 
and there are no content management shortcuts — from 
standards, content suppliers or syndication. Companies 
have seriously underestimated the effort involved in 
managing content. The barriers are many including: 
incomplete and incorrect data, a lack of content 



syndication, incomplete vendor solutions, inadequate 
solutions and services from marketplaces, multiple 
sources of content within a company and a lack of 
standards (Mitchell, 2001). Each of these factors could 
be expanded on in more detail and represent key areas to 
be addressed as companies move forward with B2B 
electronic commerce initiatives. 

CONCLUSION 

In summary, B2B e-commerce has not grown or been 
adopted as quickly as was forecast. Two primary 
reasons as explained in this paper are that marketplaces 
and exchanges did not flourish as expected and the gross 
lack of attention to, and appreciation for, the importance 
of content, which is the foundation of all commerce 
initiatives. 

More extensive research is needed to address company 
strategies and directions for successful implementation 
of B2B e-commerce and marketplace initiatives as we 
move forward in designing business models that address 
the needs of the industry and specific players in the e- 
marketplace space. 
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ABSTRACT 

Teaching advanced database applications presents opportunities and difficulties in providing students with a robust 
experience. The use of case studies, whether chosen by the instructor or by the student, is limited in scope and often 
has a finite and pre-determined termination; that is, the case studies are generally constructed to illustrate a point 
or set of points for a prescribed concept within database theory. Thus, the use of several case studies is often 
necessary. This precludes the use of larger, more difficult (and therefore more realistic) case studies due to the 
confines of semester timelines. The use of real problems, solicited from the community, can solve these challenges, 
but present others. These cases are solved in real time and imbue the student with a full and detailed grasp of the 
many factors that affect project success. This paper focuses on the experiences of students participating in an 
advanced database applications class solving challenging design problems using available technology and 
interacting with users of limited technological training. 



INTRODUCTION 

The database is the foundation of any information 
system. If the database is not robustly designed, free of 
anomalies and designed with a comprehensive 
knowledge of the organization and how the data will be 
used, any application built on top of it will contain 
problems. This leads to awkward and inefficient coding 
solutions and eventually a rebuild of the database. In 
addition, a properly designed database is a major factor 
in application speed, followed by index design and 
query design [I]. 

Database and application design does not occur within 
a vacuum. They are merely pieces of an overall and 
thorough systems analysis and design. Design 
specifications for the database are required along with 
input and output design, security and integrity. The 
stereotype of the isolated information technology 
professional is invalid. Database developers must be 



able to work in groups and communicate with non- 
technical, often untrained personnel in order to develop 
solutions that meet organizational goals. They must be 
able to translate sq)histicated concepts into workable 
solutions. They must have depth in a broad range of 
skills, both technical and interpersonal. 

Even the textbooks for courses in advanced database 
design tend to focus on one area: database theory, the 
underlying mathematical concepts, the physical 
implementation and tuning. While all these are 
important, students need an experience that will allow 
them to assimilate the entire range of a project from start 
to finish. 

Students in this course have already completed a course 
in Systems Analysis and Design and an Introductiai to 
Database Applications course, which requires 
completion of a small project. The use of case studies 
in such courses is valuable but limited. They tend to 
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illustrate one trouble spot or one application of a theory. 
Even case studies that are followed throughout a text 
suffer from the lack of interaction between diverse 
members of a project team and cannot simulate the 
organizational climate. 

Due to the confines of the standard sixteen-week 
semester, even a variety of more complex case studies 
cannot match the experience of students actually 
designing a fully functioning database application from 
start to finish. In addition, previous iterations of the 
advanced database course that is the topic of this paper 
found that students felt that completion of a “toy” 
project was unfulfilling. The amount of effort that was 
expended for a throwaway project left students 
dissatisfied. 

PROJECT 

A decision was made to obtain real projects for the next 
group of advanced students. Since Wentworth Institute 
of Technology is located within an economically 
disadvantaged section of Boston, a philosophical 
decision was made that only projects from local non- 
profit agencies would be solicited. 

This requirement lead to the first set of difficulties. 
Non-profit agencies are always underfunded and may 
not have the necessary equipment. Social service 
agencies also tend to lack technically astute personnel 
due to their high cost. The time necessary to sit with a 
student design team would be time away from the 
agency’s mission. 

In addition, the projects chosen had to be completed 
within a sixteen-week semester. Information technology 
proj ects are organizationally famous for being over time 
and cost estimates. This meant that the projects had to 
be small enough to finish — in fairness to both the 
students and the chosen agency — but still complex 
enough to challenge the students. 

A previous attempt at finding real projects for senior 
design students had been attempted at Wentworth. It 
was abandoned due to problems with the agencies 
chosen not wishing to expend the necessary time. 
However, the projects in question were not non-profit 
organizations, mostly small businesses. It was hoped 
that the incentive of decreased time spent on automated 
functions would be motivation for the non-profit 
agencies’ staff 



There was then the difficulty of identifying appropriate 
agencies and calling them with this proposal. Since 
Wentworth is involved with a program entitled, 
“Technology Goes Home,” whereobsolete equipment is 
donated to local agencies, there were the names of local 
agencies that had requested equipment available. This 
significantly cut down on cold calling of agencies. 

Several meetings were necessary with the course 
instructor and the acting department chair, along with 
representatives of two agencies that had agreed to be a 
part of the pilot program. The two agencies were the 
Parker Hill/Fenway branch of Action for Boston 
Community Development (ABCD), and the Nelson 
Mandela Training Center. During this time, ten projects 
were identified and carefully negotiated to be 
technically and operationally feasible. 

The ABCD projects consisted of a database of daycare 
openings, searchable by location, languages spoken, 
ages accepted, and fee schedule; a database of 
employment opportunities searchable by job title, 
location, and requirements; a database of education 
training programs; a database of available low-income 
housing, searchable by location, cost, size and subsidies 
accepted; and a grant-tracking database to assist the 
agency in coordinating requests for external funding and 
meeting grant guidelines. 

The Nelson Mandela Learning Center wound up with a 
centralized client database and five applications: a 
marketing application for students completing programs 
who may want to take an advanced course; an 
application that tracks student progress through class 
levels, one for computer students and one for accounting 
students; an online test bank that randomly serves 
questions to students taking exams (to discourage 
cheating) that automatically calculates their grades; and 
an online application that can be filled out over the 
Internet for new students. 

The agencies had computers that were capable of 
running a database application, but only Access 2000 
software was available. This precluded the students 
from gaining experience in SQL Server or ORACLE, 
two highly complex and popular database engines. 

Students were surveyed during the first week of class. 
There was one questionnaire on student skills outside of 
prescribed courseworkin the curriculum, specifically if 
they knew how to web-enable databases. The second 
questionnaire described eachproject, and asked students 
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to rank them in the order of their interest. The instructor 
then spent several hours matching student skills to 
interests and assigning groups. All but one groip 
received their first or second choice of project. 

During the next sixteen weeks, the instructor acted as a 
project manager. It was felt that the agencies all 
deserved a completed project after the time invested, so 
it was imperative to monitor groip progress more 
strenuously than when teaching the course with toy 
projects. 

Students were often frustrated with the limited ability of 
agency staff to answer specific questions or grasp 
technical concepts. Then, as the projects came together 
and the agencies began to see the potential, feature creep 
became intense. In addition, some agency contacts were 
under the misconception that students would be doing 
data entry of existing paper records. The instructor had 
to intervene on more than one occasion. 

During the final weeks, when the projects were actually 
being coded, students discovered a known bug in a 
service patch of Access 2000. There was no time to 
obtain and install the fix, so students were forced to 
write workarounds. 

At the end of the semester, all ten projects wa*e fully 
functional. Students rq^orted an estimated 500 
programming hours had been donated to the two 
agencies. Students gave professional presentations on 
their projects and demonstrated the software for their 
agencies to representatives of Wentworth and the 
participating agencies. Then, students met at each 
agency for the installation and staff training. 

PREVIOUS RESEARCH 

Service learning is not a new concept, and there are 
other ways to introduce it to students. Other 
organizations have various implementations of 
providing a real-world experience to students. 
Wisconsin University at Eau Claire requires all 
baccalaureate students to complete thirty or more clock- 
hours engaged in an approved service-learning project, 
to allow students to apply classroom knowledge and 
serve their community [2]. All twenty-three campuses 
of California State University have a designated service- 
learning center. Service learning has been integrated 
throughout their curriculum, and they have found their 
programs to be highly successful [3]. Carnegie Mellon 
offers credit to students through a course specifically for 



computer science majors, entitled “Service Learning in 
Technology.” Students are required to work with a non- 
profit organization [4]. 

SOME OBSERVATIONS 

Programs that offer students real projects are not to be 
undertaken lightly. Many months of preparation were 
required from the instructor, and many hours of extra 
work were added as negotiations between students and 
agencies, or disagreements between student groups 
arose. 

One student group repeatedly insisted that they should 
be compensated financially. The instructor replied that 
a project was required anyway, and that the real project 
would be advantageous on their resumes and could 
realistically translate into a better post-graduation job 
offer. 

The student frustration with the users was actually seen 
by the instructor as a benefit. Interacting with non- 
technical users is an unavoidable feature of information 
projects, and the more experience with it, the better. 
Communications skills are imperative to designing a 
system that does what the customer wants. 

Overall, student enthusiasm was quite higfi. Students 
were very animated in laboratory sessions and were 
clearly pleased when the agency users expressed 
satisfaction with project progress. Since some projects 
for each agency required a common database, students 
got to see first hand the intricacies of designing large- 
scale databases and learned to work with other students 
not in their group. 

The director of the Nelson Mandela Training Center 
gave students an eloquent and impromptu speech on the 
impact of their work on their agency. He stated that in 
a time when social service agencies are underfunded and 
understaffed and needs are exploding, the software 
would free the staff from repetitive tasks and speed the 
services to clients. The students were moved. 

Other non-quantitative observations are that some 
students were not performing as team members. One 
student was reported for refusing to repair significant 
anomalies in his data design that were causing problems 
in another group’s application. This also required 
instructor intervention. These problems occur in all jobs 
and students need to learn to address them. 
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Another added benefit is the guarantee that student work 
is original. These custom-designed projects preclude 
the use of solutions copied from other students, 
textbooks or web sites. This is especially true when 
using non-profit organizations where the usual business 
models of customers, orders, products, and inventory 
aren’t applicable. 

FEEDBACK BY AGENCIES 

Both the Nelson Mandela Training Center and ABCD 
expressed satisfaction with the process and the result 
They would recommend working with Wentworth 
students to other agencies, and expressed a desire to 
work with Wentworth again. 

The director of the Nelson Mandela Center has already 
requested a web-enabled online survey to measure 
student satisfaction and tailor program features to client 
needs. This would work with the core demographics 
portions of the existing database. The director of Parker 
Hill/Fenway ABCD has expressed interest in replicating 
the software to other divisions. 

FEEDBACK BY STUDENTS 

The following are comments from student evaluations 
and instructor interviews: 

1. We need to be challenged more; this is a good 
project. 

2. We should have done a project in SQL Server or 
ORACLE; we don’t use Access in my job and the 
DBA (database administrator) laughed at me when 
I told him we were learning Access. 

3. Next time, don’t have a real project. 

4. This took WAY more time than I expected (the 
most common obsawation from students). 

FUTURE PLANS 

The instructor plans on continuing to use real projects in 
the classroom. The benefits seem to greatly outweigh 
the difficulties and further iterations will eliminate 
problems experienced in this pilot stucfy. The non-profit 
agencies win; the students win; and Wentworth also 



wins as it proves its commitment to the community. The 
department head has offered continuing support. 

It is hoped that as word of the program expands, that 
area agencies will contact Wentworth instead of the 
previous tedious process. 

The department is currently exploring partnering with a 
local college in the Boston area that offers an education 
major. The other school would devise an educational 
assessment study to determine if students in the sections 
using real projects have a greater understanding and 
retention of the material. 

This program could also be expanded to include other 
majors at Wentworth. It is currently being explored to 
have the Management of Technology majors formally 
act as project managers so that they can benefit from a 
live project. This would free the instnictor from project 
management duties as well. 

It is also hoped that projects can be completed using 
something other than Access. While Access is an 
affordable database, it lacks the sophistication of higher 
end databases and will not assist students in obtaining 
more lucrative positions such as database developers or 
database administrators. 

CONCLUSIONS 

Using real projects is preferable in advanced classes that 
require students to draw on a significant and broad pool 
of resources cultivated in prerequisites. This will 
prepare students to be more successful in their jobs after 
graduation and expand their resumes. It is also hoped 
that the experience of working with nonprofit agencies 
will instill in students a lasting commitment to 
community service. 
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TOPIC 

Within the last five years the number of universities and 
colleges offering courses online has increased 
substantially; however, there is some skepticism about 
the actual value of online learning. The current state of 
online learning evaluation, its adequacy, and potential 
for additional evaluation methods will be discussed by 
the panel presenters. 

This panel presentation is intended to foster dialogue on 
the importance and need for online learning evaluation 
and on the comparison of online learning environments 
with the influx of a variety of Virtual Learning 
Communities (VLCs) including Blackboard, WebCT, 
and eCollege. It is expected that the panel presentation 
and dialog will generate ideas for research exploring the 
effectiveness of online learning in these VLCs. 



FORMAT 

Each panelist will begin with a brief overview of his/her 
experiences with on-line learning using one or more of 
the Virtual Learning Communities (VLC’s) described 
above. The audience will then be invited to ask 
questions from the panel members regarding their 
experiences and thoughts. 

Audience members will take with them a better 
understanding of the most common VLC’s in use today, 
their strengths and weaknesses, as well as general 
guidelines for how to use these tools pedagogically in 
the classroom. 
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ABSTRACT 

This study compared the effects of using voice mail (v-mail) to electronic mail (e-mail) over the Internet to provide 
student feedback using Media Richness Theory (MRT) and Social Presence Theory (SPT) as the theoretical 
framework. MRT and SPT would predict that v-mail would be perceived as higher than e-mail in media richness and 
social presence. Results indicate, however, that while v-mail was perceived to have significantly higher social 
presence, the two media were not significantly different in terms of perceived mediarichness. Both e-mail and v-mail 
were perceived as capable of providing a reasonably high quality of feedback. Implications of these findings are 
discussed. 



INTRODUCTION 

The rise of Internet computing and related technologies 
has created a technological platform that allows 
professors to rethink the way they deliver feedback to 
their students. Providing timely and appropriate 
feedback is a must for a student to have a successful 
learning experience (Nabors, 1999). It can promote 
student learning, increase student satisfaction with a 
course and an instructor, and improve course/program 
completion rates (Hackman & Walker, 1990; Inglis, 
1998). Students receive intrinsic feedback while 
engaging in their learning activities and/or extrinsic 
feedback through some form of communication with the 
instructor (Inglis, 1998). Traditionally, the extrinsic 
feedback is provided either through face-to«face or via 
written comments on student assignments and exams 
during the following class meeting or much later. 
Computer-mediated-communication (CMC) technology 
has made it possible for students to receive their 
feedback electronically without having to wait for a 
meeting with the instructor at a specified time or 



location. This can reduce the time between submitting 
an assignment and receiving feedback on the assignment 
(Inglis, 1998; Xu, 1996). 

Due to its ability to support asynchronous 
communication, text-based e-mail facilitates 
communication between individuals or groups of 
individuals (El-Shinnawy & Markus, 1997) and has 
become a popular communication medium (Inglis, 
1998). In a recent study, Blake (2000) noted that most 
students preferred submitting an assignment as well as 
receiving feedback via the Internet. In comparing the 
Internet with face-to-face feedback from the instructors, 
more students rated receiving feedback through the 
Internet as extremely effective (64.6% vs. 54.2%) or 
very effective (27.1% vs. 16.7%). The increased use of 
e-mail among both students and faculty creates new 
avenues for delivering feedback to students. Many 
faculty members have used e-mail to communicate with 
their students, thus providing immediate feedback and 
a direct means fa* communication (Huang, 2000; 
Nabors, 1999; O’Neill, 1997). 
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Bates (1995, p. 202) noted that CMC is “one of the 
fastest growing technologies, in terms of the number of 
teachers and learners who are using it.” E-mail is one 
type of CMC that allows for both one-to-one and one-to- 
many textual communications via a host computer. The 
recipient can save the e-mail messages indefinitely in 
computer memory, read or reread them at any time, 
copy them, change them, or forward them. E-mail 
provides faster communication with greater efficiency 
compared to face-to-face communication or postal 
services. E-mail allows individuals to exchange ideas 
spontaneously and casually, discuss problems and share 
skills, coordinate activities, and stay in touch with one 
another without regard to an individual’s physical 
location (Grunwald, 1990; Huang, 2000; Sproull & 
Kiesler, 1991). 

In spite of the benefits that e-mail can offer, it is a 
limited symbolic representation system devoid of 
oratory and graphic appeal. It requires keyboard skills 
and writing abilities in order to avoid misunderstandings 
(Bates, 1995; Sproull 8c Kiesler, 1991). Social and 
contextual cues that usually regulate and influence 
group communication dynamics are missing or 
attenuated. However, when there is a highly developed 
and shared interpretive context, e-mail is an appropriate 
communication mode (Huang, Watson, & Wei, 1998; 
Trevino, Lengel, 8c Daft, 1987; Zack, 1993). It can be 
used to complete numerous instructional activities such 
as course material presentation, collaborative and 
project work, help hotline, group discussion and group 
activities, and evaluation (Xu, 1996). 

While many faculty members have embraced traditional 
e-mail as a means of communicating with students, few 
have ventured beyond text-based communication such 
as using v-mail systems to communicate with students 
via voice rather than the text form of e-mail (Huang, 
2000). The purpose of this research was to explore the 
media richness and social presence of the two media, 
traditional text-based e-mail and internet enabled v-mail, 
in providing feedback to students, and to compare which 
of the media, traditional e-mail or internet enabled v- 
mail, would provide a higher quality of feedback to 
students. Media Richness Theory (MRT) and Social 
Presence Theory (SPT) provide the theoretical 
foundation for comparing the two media. These two 
theories suggest that aie medium may be inherently 
richer or more socially present than another. While 
there has been considerable research on MRT and SPT, 
and measures have been develq^ed to assess the media 



richness and social presence constructs, there has been 
little, if any, attempt to reconcile these two constructs, 
or to apply them within an education and learning 
context. Therefore, in conducting this study, the 
following research questions were addressed: 

RQ#1) Would students perceive v-mail attachments 
sent over the Internet to have a higher media 
richness as compared with more traditional text- 
based e-mail? 

RQ#2) Would students perceive v-mail sent over the 
Internet to have a higher social presence as 
compared with more traditional text-based e- 
mail? 

RQ#3) Would students perceive v-mail sent over the 
Internet to provide a higher feedback quality as 
compared with more traditional text-based e- 
mail? 

BACKGROUND AND LITERATURE REVIEW 

Effective communication has long been recognized as a 
key element in problem solving and decision making 
within and among organizations. The emerging 
information technologies, such as e-mail and v-mail, 
have expanded communication choices. MRT and SPT 
have been used to examine the effectiveness and 
appropriateness of traditional media, such as face-to- 
face meetings and written documents, as well as new 
media, such as e-mail and v-mail, in organizational 
communication. 

Media Richness Theory 

According to MRT, different media vary in information 
richness based on their capability of providing 
immediate feedback, the number of cues involved (i.e., 
body language, facial expression, tone of voice, etc.), 
message personalization, and natural languages (Daft & 
Lengel 1984; 1986). The more attributes the medium 
possesses, the richer the medium. Based on this 
definition of media richness, face-to-face 
communication is considered to be the richest medium 
because it provides immediate feedback and multiple 
cues, as well as utilizes natural languages. This is 
followed in media richness by the telephone which has 
fast feedback capability but lacks visual cues; 
“individuals have to rely on language content and audio 
cues to reach understanding” (Daft & Lengel, 1984, p. 
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1 98). Formal written communication is considered even 
less rich because of slow feedback, limited visual cues, 
and lack of audio cues. 

Based on MRT one would predict that e-mail (which is 
an asynchronous text-based medium with near 
instantaneous delivery speed) would fall somewhere 
between synchronous voice communication and formal 
written communication (e.g., a letter sent in the mail) in 
terms of media richness. A study by Zmud, Lind, and 
Young (1990) provides some empirical support for this. 
Several studies have shown empirical support for the 
notion that voice communication (telq)hone) is 
perceived as having higher media ridmess than e-mail 
(Schmitz & Fulk, 1991;Fulk&Ryu, 1990; and Trevino 
et al., 1990). While e-mail is text-based (low language 
variety and cues) and asynchronous, it offers faster 
communication than paper-based documents, which 
allows for immediate feedback. Therefore, e-mail has 
been ranked between face-to-face meetings and letters 
in terms of its perceived media richness (Trevino, et al., 
1990; Trevino, Webster, & Stein, 2000). 

MRT would predict that v-mail (which allows for 
asynchronous voice communication) would fall 
somewhere between the telephone (which allows 
synchronous voice communication) and e-mail. Again, 
the study by Zmud, Lind, and Young (1990) provides 
some empirical support for this, as does a study by Rice 
(1992). 

Other studies (Dennis & Kinney, 1998; Trevino, 
Webster, & Stein, 2000; Westmyer, DiCioccio, & 
Rubin, 1998) have confirmed that iov traditional media 
(i.e., face-to-face, telephone, letters, memos), perceived 
media richness was significantly higher with increased 
multiplicity of cues and increased immediacy of 
feedback. However, with respect to new media such as 
e-mail, there has been some debate as to whether or not 
the medium itself is inherently lean. Studies by Zack 
(1993) and Lee (1994), for example, suggest that e-mail 
can be regarded as a rich medium, particularly when 
those who are communicating have a shared context or 
understanding. These authors and others would argue 
that media richness is not an inherent property of the 
medium, but rather an “emergent property of the 
interaction of the electronic-mail medium with its 
organizational context’’ (Lee, 1994, p. 143). Even 
proponents of media richness theory have acknowledge 
that e-mail, because of its rapid delivery, maybe quite 
appropriate for exchanging time-sensitive information, 
and in this respect it is similar to the traditional medium 



of the telephone (Rice, 1993; Rice & Case, 1983; 
Trevino, Lengpl, & Daft, 1987). 

Social Presence Theory 

According to Short, Williams, and Christie (1976), 
social presence is a subjective quality of the 
communication medium and relates to the social 
psychology concepts of intimacy (determined by 
physical distance, eye contact, smiling, and personal 
topics of conversation) and immediacy (determined by 
the media capacity of transmitting information). 
Therefore, social presence depends on the amount of 
information transmitted as well as the words conveyed 
during communication, verbal cues (e.g., tone of voice), 
nonverbal cues (e.g. facial expression, direction of gaze, 
posture, dress), and the communication context. Based 
on this theory, communication media such as face-to- 
face meetings, which are capable of conveying 
nonverbal cues and social context cues, are considered 
to have higher social presence than computer-mediated 
communication media and written documents because 
they lack nonverbal feedback cues (King & Xia, 1999). 
Presumably, the higher intimacy and immediacy the 
medium has, the richer the medium and the higher the 
social presence (Daft & Lengel, 1984; 1986; Dennis & 
Kinney, 1998; Short, Williams, & Christie, 1976; 
Trevino, Webster, & Stein, 2000). 

Rice (1993) used social presence theory to compare 
traditional and new media by analyzing data from six 
studies designed to examine the use and effects of new 
media. He found that due to the lack of social presence, 
both V-mail and e-mail were ranked lower in their 
overall task appropriateness than traditional face-to-face 
meetings and e-mail was ranked even lower than v-mail 
in both overall ^propriateness and for exchanging 
timely or confidential information. 

Based on MRT and SPT, one wouldexpect text-based e- 
mail to be a richer and more socially present 
communication medium for providing feedback to 
students than the written feedback students normally 
receive on their returned assignments and exams 
(because e-mail has the ability to provide immediate 
feedback). However, e-mail would be predicted to be a 
much leaner and less socially present medium than 
traditional face-to-face or telephone communication due 
to its inability to provide the same level of social 
presence and nonverbal cues. In the same vein, MRT 
and SPT would predict that v-mail via the Internet 
should be perceived as a richer and more socially 
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present medium than text-based e-mail in providing 
feedback because v-mail has a personal focus and is 
easier to express in (tynamic natural language, whereas 
e-mail can only convey static visual cues in text. 

While social presence and media richness are treated as 
separate and distinct constructs in the literature, they 
would appear to be quite closely related constructs. 
There have been no studies of which the authors are 
aware that measure both constructs and seek to compare 
the results. Thus, although one might predict that a 
medium which is perceived to have higher media 
richness would also be perceived as having a higher 
social presence (and vice versa), this has not been 
demonstrated empirically. 

Moreover, no single study has been conducted to 
compare the effects of using Internet based v-mail to e- 
mail to provide feedback to students. Recently, internet- 
based communication technologies have made it 
possible for individuals to send and receive 
asynchronous voice communication over the Internet in 
the form of e-mail attachments more easily. This study 
was designed to compare students’ perceived media 
richness, social presence, and the quality of feedback 
associated with v-mail sent over the Internet versus e- 
mail. 

RESEARCH METHOD 

To address the research questions of this study, 
traditional text-based e-mail and Internet-based v-mail 
were utilized to provide feedback to students. In this 
study, the traditional text-based e-mail feedback was 
composed, delivered, and received using standard e-mail 
software whereas the v-mail feedback was recorded and 
played back using PureVoice™. PureVoice™ software 
allows the user to share his or her voice file with anyone 
in the world who has an e-mail account, access to the 
Internet, and a computer running on either the W indows 
or the Macintosh operating system. PureVoice^^ is 
compatible with most e-mail systems, including 
Qualcomm’s Eudora Pro and Eudora Light. The 
PureVoice™ Player-Recorder lets the user record and 
send voice messages as e-mail attachments and allows 
the recipient play back the sender’s voice-mail messages 
on his or her Windows or Macintosh computer with the 
click of a mouse. Voice messages composed with 
PureVoice^^ have extremely high sound quality — 
(nearly as high as one would experience in a local phone 
call made over a hi^-quality connection). Yet, because 
PureVoice^^ uses compression technology, the 



messages take up relatively little disk space and can be 
sent very quickly over the Internet (Qualcomm 
Incorporated, 1998). 

Design 

An experimental design was employed in this study. 
Students enrolled in two graduate courses at a large 
southeastern university during the 1999-2000 academic 
year were randomly assigned to one of the two 
experimental conditions. Internet-based v-mail (v-mail 
group) and traditional text-based e-mail (e-mail group). 
Students in the v-mail group received their feedback via 
an attached voice file sent over the Internet as an e-mail 
message. Students in the e-mail group received their 
feedback over the Internet using a traditional text-based 
e-mail message without a voice attachment. Such an 
experimental design ensured that students not assigned 
to the treatment group would not pay a penalty in terms 
of how quickly they received feedback on their exams 
from the instructor. A total of 46 students participated in 
the study with 23 students being assigned to the v-mail 
group and 23 students being assigned to the e-mail 
group. 

Students in both groups received either voice-based or 
text-based feedback on their exams in the course. A 
copy of the exam questions was attached to each e-mail 
message, regardless of the group to which a student was 
assigned. When preparing an e-mail feedback message, 
the instructor composed and edited using a word 
processor whereas a voice feedback message could only 
be edited by erasing the message and starting over again 
(El-Shinnawy & Markus, 1997). In order to provide 
consistent and high quality feedback to both groups, all 
feedback for each student was first composed and edited 
in a word processor, then e-mailed to students in the e- 
mail group or read to those in the v-mail group. This 
extra step of composing and editing ensured that 
students received the same level of feedback and 
comments regardless of the group to vdiich they were 
assigned. 

Data Collection and Analyses 

Students in both groups completed an anonymous 
questionnaire with each of the two versions tailored to 
the particular group in which the students were assigned 
(v-mail or e-mail group). Some students offered 
additional comments in response to open-ended 
questions that were included in the questionnaire. Apart 
from minor modification of wording to reflect the 
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treatment group to which a student was assigned, the 
two versions of the survey were designed to be as 
identical as possible and to allow for comparing 
responses across the two treatment groups. 

The questionnaire included established measures (4-item 
scales) for media richness (Schmitz & Fulk, 1991) and 
social presence (Short, Williams, & Christie, 1976), as 
well as specifically designed items measuring perceived 
quality of the feedback (see Appendix). The media 
richness measure was adapted from Schmitz and Fulk 
(1991) and was based on the criteria specified by Daft 
and Lengel (1984). This measure has been widely used 
by others (Carlson & Zmud, 1999; El-Shinnawy & 
Markus, 1997; Fulk & Schmitz, 1995). The social 
presence measure was based on an instrument developed 
by Short, Williams, and Christie (1 976). In addition, the 
questionnaire included 4-item measures for perceived 
feedback quality, single-item measures for perceived 
usefulness and perceived ease-of-use of the software, as 
well as several open-ended questions designed to gather 
qualitative data from the students regarding their 
reactions to the feedback mechanism used in the group 
to which they were assigned. The ^pendix holds the 
measurement items that were used to assess media 
richness, social presence, feedback quality, usefulness, 
and ease-of-use. 

Both descriptive and inferential statistical analyses were 
conducted to describe and compare the perceptions of 
media richness, social presence, quality of feedback, and 
the usefulness and ease-of-use of the software. 
Qualitative responses were discussed and related to the 
quantitative data. 

RESULTS 

Though the items used to assess media richness and 
social presence represent established measures, their 
reliability was assessed using Cronbach’s alpha. The 
four-item measure for media richness had a reliability of 
0.61. Statistical analysis revealed that the reliability of 
this scale could be increased to 0.72 by dropping the 
first item while retaining the other three items. A 
decision was made, however, to retain all four items, 
since they represent an established measure that has 
been used elsewhere (Carlson & Zmud, 1999; El- 
Shinnawy & Markus, 1997; Fulk & Schmitz, 1995; 
Schmitz & Fulk, 1991) and since their reliability was 
judged to be adequate for exploratory purposes. The 
four-item measure for social presence had areliability of 



0.83. The four items designed to measure perceived 
feedback quality were factor analyzed and revealed a 
single-factor structure. These four items, which 
measured helpfulness, usefulness, quantity and detail of 
the feedback received, exhibited high reliability 
(Cronbach’s alpha = 0.86). 

A one-way analysis of variance (ANOVA) indicated that 
there was a significant difference between treatment 
groups on perceived social presence of the media (F = 
28.65, Sig. = 0.000). Specifically, attached v-mail files 
sent over the Internet (using PureVoice^*^) were 
perceived as having significantly higher social presence 
as compared with text-based e-mail. A similar one-way 
ANOVA was conducted to investigate whether the same 
pattern of results would hold true for perceived media 
richness. This ANOVA, however, revealed no signifi- 
cant difference between treatment groups in terms of 
perceived overall media richness (F = 0.428, Sig. = 
0.517). 

Both media (e-mail and v-mail) were perceived as being 
reasonably rich and relatively high in social presence 
(mean score 3.74 vs. 3.86 and 3.25 vs. 4.08, 
respectively, on a 5-point scale with 1 being the lowest 
richness/social presence and 5 being the highest). As 
expected, both e-mailand v-mail were perceived to have 
provided relatively high quality feedback (mean score 
3.70 vs. 3.76 on a 5-point scale with 1 being the lowest 
quality and 5 being the highest). The difference between 
the two groups was not found to be statistically 
significant in the one-way ANOVA (F = 0.097, Sig. = 
0.756). 

The e-mail group perceived that the software they used 
was both more useful (4. 1 mean) and easier to use (4.74 
mean), as compared with the v-mail group on these 
same measures (3.95 mean for usefulness and ease-of- 
use). The difference in perceived usefulness of the 
software was not found to be significant (F = 0.42 1 , Sig. 
= 0.502), but the difference in perceived ease-of-use 
found to be statistically significant (F = 8.90, Sig. = 
0.005). 

Tables 1 and 2 provide a representative sampling of the 
qualitative data students furnished on the survey in 
response to the open-ended question, “What was the 
thing that you liked the most about receiving feedback 
via e-mail/attached voice-mail files?” and “What was 
the thing that you liked the least about receiving 
feedback via e-mail/ attached voice-mail files?” 
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TABLE 1 

REPRESENTATIVE RESPONSES TO THE QUESTION, 

“WHAT WAS THE THING THAT YOU LIKED THE MOST ABOUT 
RECEIVING FEEDBACK VIA E-MAIL/ATTACHED VOICE-MAIL FILES?” 



E-mail Group V-mail Group 

Could be printed and filed/shared. Could easily refer to It was good to hear comments and be able to form an impression of the 
earlier/later sentences within the e-mail. instructor’s view/expectation of work. 



I like the real voice from instructor. From that, I know that instmctor 
really read/graded my paper. Otherwise, I really wonder/doubt that the 
instructor read it or not. 

It attached a human to the exchange. Also I believe, it gave a more 
honest evaluation in the sense that I did not interpret correctly or 
incorrectly what the words mean (i.e., inflection, posture, regulative). It 
was right there. 



It gave me sometime to reflect on the answer. Also, it was useful 
in providing feedback and questions to the instructor. 

Timely: can get the feedback before the next class. Personal: 
feedback is based on my own performance and more focus, more 
specific. Easy to reference: since it is a written message^ it’s very 
easy to use as a reference. 

The rapidity and completeness. I liked knowing what I got on the 
exam and why before I got the exam back. 

Easier to read than on a copy, usually more detailed — having it 
in another place than on a copy force you to think and retrace 
what you wrote. 

The timeliness of the feedback — getting results in private. More 
personal than class review. 

E-mail feedback is descriptive and tailored to individuals 

Getting feedback before class. . . . More one-on-one feedback than 
class time. 

I liked the timely nature and availability. It is nice to be able to 
reference the comments. 



It is really to identify thepoints that you wanted to stress. The inflection 
of your voice. 



I felt like you are probably more familiar with us, the students, and that 
you spoke to each of us “on our level.” 



Psychological. Make me feel differaitiated from the class. Easy to 
understand than to read someones handwriting. 

The potential timeliness of feedback. Ability to say what you mean may 
be quicker and easier than writing. 



Tailored, customized and more personal. 



I think it allowed for more timely response. Forming an answer and 
recording is faster than writing it out and seems to allow the responses 
to be returned to us faster. 



TABLE 2 

REPRESENTATIVE RESPONSES TO THE QUESTION, 

“WHAT WAS THE THING THAT YOU LIKED THE LEAST ABOUT 
RECEIVING FEEDBACK VIAE-MAILmA ATTACHED VOICE^MAIL FILES?” 



V-mail Group 

Not readily referable. Have to play back several times if you don’t 
understand part of points. 

With the voice responses it is not as easy to move backwards and 
forwards in the response to pinpoint and coordinate with each section of 
the test. I had to listen to the response all the way through. It was not 
always easy to get to the right place. 



E-mail Group 

No info written on test. If I look a test in the future I will have to 
find the e-mail message. 

I think of the two forms of written communication e-mail & 
handwritten on exam, the latter is richer. Professors tend to write 
little notes next to item, from which I infer more than summaries 
of grading of my answers. 

Not having my answers to compare to comments/grading. 
Scanned test with answers would be ideal attachment, but not 
practical for large class. 

Thecomments could not be specifically related to answers on the 
exam. You are not able to see a comment next to a specific area. 

The e-mail comments were too general. 

I didn’t have my exam back yet, so couldn’t remember exactly 
what I wrote to compare it to the feedback. 



Making myself take the time to do it. Using the normal method was 
more convenient since I am in class and everyone else is reading over 
their test at the same time. 



The initial setup of the plug-in. A URL for the plug-in with the e-mail, 
or, a pilot e-mail with plug-in URL to test sample would have been 
good. 

In transmitting the message or information, it isn’t much different from 
e-mail in usefulness and effectiveness 

If you did not do well, it feels likethe professor is in front of you telling 
you the bad news. 
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As the comments in the two tables reveal, students in 
both treatment groups appreciated the timely and 
personalized feedback. Although students in both groups 
had copies of the exam questions, both groups found it 
frustrating to relate the feedback to their own exam 
answers. This frustration was caused by two factors: (1) 
students received feedback before they received the 
hardcopy of their exam answers, and (2) the feedback 
was not on the exam itself. Students in the v-mail group 
experienced additional frustrations in that they had to 
install the PureVoice^^ software and had to replay the 
entire message multiple times in order to access specific 
portions of the feedback. It is clear, however, that 
students in the v-mail group appreciated the higher 
social presence of the medium, whereas students in the 
e-mail group appreciated receiving the text-based 
feedback because it could be printed, filed, shared, used 
as a reference and accessed in a non-sequential (direct or 
random) manner. 

DISCUSSION AND IMPLICATIONS 

Perceptions of social presence observed from this study 
were consistent with SPT. As SPT would predict, 
attached v-mail files sent over the Internet were 
perceived as having a higher social presence than e- 
mail. In accordance with MRT, v-mail was perceived to 
be slightly higher than e-mail in terms of media 
richness. However, this difference was not found to be 
statistically significant, and in fact e-mail (wiiich is 
often regarded as a lean medium) was perceived as 
being a reasonably rich communication medium. This 
suggests that the two constructs — media richness and 
social presence — are somewhat separate and distinct. 

Although the difference in overall media richness 
perceptions was small and statistically insignificant, it 
was noted that e-mail was perceived to be significantly 
richer in “giving and receiving timely feedback” than v- 
mail (mean score 4.57 vs. 3.96 on a 5-point scale with 1 
being the lowest richness and 5 being the highest, F = 
10.08, p = .003). This might have affected the 
magnitude of the overall media richness perception 
because such a significantly different richness 
perception occurred in the opposite direction to the other 
three items measuring the overall media richness 
perception. 

Perceived media richness may have also been affected 
by the perceived ease-of-use and previous knowledge of 
and experience with the software. Based on the 
experimental design, both groups physically received 



their feedback via e-mail or v-mail at the same time, but 
they perceived the timeliness of the media differently. 
This could be explained by the fact that individual 
perceptions of media richness are infiuencednot only by 
objective media characteristics (Rice, 1993; Trevino, 
Webster, & Stein, 2000), but by individual’s experience 
with the medium (Carlson & Zmud, 1999; Fulk & 
Schmitz, 1995; King & Xia, 1999; Schmitz & Fulk, 
1991), interaction between the individual and 
technology (media experience, preference, and skill) 
(Rice, 1992; Trevino, Webster, & Stein, 2000), 
perceived usefulness (Fulk & Schmitz, 1995), and 
perceived ease of use (the extent to which an individual 
believes that using a particular technology system would 
be free of effort (Davis, 1989; Davis, Bagozzi, & 
Warshaw, 1989; Trevino, Webster, & Stein, 2000). 

In this study, all of the subjects were graduate students 
who had extensive experience with e-mail and Internet 
usage, but no one had prior experience of using v-mail 
software. Although PureVoice^*^ was easy to use, it was 
new software and had to be installed for the first time 
for the students in the v-mail group. This not only 
affected an individual’s media experiences and 
expertise, but may have also influenced the perceived 
ease-of-use (e-mail software was perceived to be 
significantly easer to use than v-mail software, F = 8.90, 
Sig. = 0.005). This might have influenced students’ 
perception that e-mail was a significantly richer medium 
in “giving and receiving timely feedback” than v-mail. 
The qualitative data collected provides an indication that 
at least some students found the initial setup of the 
PureVoice^^ software (downloading/installing the plug- 
in) to be somewhat inconvenient. In contrast, students 
in the e-mail group (who presumably already had access 
to e-mail software) would not have had to deal with 
downloading and installing any special software. 

Putting the issue of timeliness aside, it is still possible 
that V-mail simply may not be perceived as a richer 
medium than e-mail in providing feedback to students 
via the Internet. For the other three items measuring the 
overall media richness (“transmitting a variety of 
different cues beyond the spoken message,” “tailoring 
messages to your own or other personal characteristics,” 
and “using rich and varied language”), v-mail was 
perceived as a somewhat richer medium than e-mail. 
Such differences, however, did not reach statistical 
significance (p> .05). 

The fact that v-mail was not perceived to be 
significantly higher than e-mail in terms of media 
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richness appears to run counter to MRT. However, it is 
consistent with some research studies that have 
suggested that e-mail can, in fact, be a rich medium 
(Lee, 1994; Zack, 1993). 

Both the e-mail group and the v-mail group perceived 
feedback quality to be reasonably high and there was no 
significant difference between the two groups. This 
indicates that the step of composing and editing 
feedback for each student beforehand did control for the 
quality of feedback. In addition, this result suggests that 
perceived feedback quality may be influenced less by 
the type of medium that is used, and more by the actual 
content of the feedback. This study did not compare the 
quality of feedback provided by these two new media 
with that provided by more traditional media such as 
paper-based or face-to-face. Therefore, it is impossible 
to say that either e-mail or v-mail produces higher 
quality feedback than the traditional media do. One 
advantage of providing text-based or voice-based 
feedback to each student individually via the Internet 
was that it avoided student complaints of embarrassment 
or invasions of privacy noted by Blake (2000). In this 
study, no students expressed their dissatisfaction in this 
regard and at least one student mentioned “getting 
results in private” as one of the things that s/he liked the 
most. 

Students in the v-mail group experienced frustration 
because they could only access portions ofthe feedback 
in a sequential fashion (by playing back the entire 
message) as compared with a text-based message that 
can be more easily scanned in a direct access (non- 
sequential) fashion. This finding is consistent with the 
literature (El-Shinnawy & Markus, 1997; Valachich, 
Paranka, George, & Nunamaker, 1993). E-mail’s text 
quality helps people interpret the message accurately, 
and it is easier to process, filter and transfer than v-mail. 
A recipient of e-mail messages has random access to 
any of the messages in his/her inbox and the recipient 
can scan through a message in order to get to the 
important points, whereas a v-mail recipient has to go 
through all the v-mail messages sequentially and if there 
is a long sequence of messages, the cognitive overload 
tends to be higher. 

Both e-mail and v-mail appear to be useful media for 
receiving feedback from the instructor and both are easy 
to use. In this study, both e-mail and v-mail received a 
high mean score (4. 1 3 and 3.95 respectively on a 5-point 
scale with 1 being the lowest usefulness and 5 being the 
highest). This perceived high level of usefulness might 



relate to the relatively high quality of feedback students 
perceived because students in both groups ranked 
quality of feedback above 3.70 on a 5-point scale with 
1 being the lowest quality and 5 being the highest. With 
regard to the ease-of-use perception, e-mail was 
perceived to be significantly easier touse than voicemail 
(presumably because v-mail was new to the students and 
required them to download software for the initial 
installation). 

LIMITATIONS AND 

DIRECTIONS FOR FUTURE RESEARCH 

As with all studies, this one is subject to certain 
limitations. First, while useful data were gathered to 
assess media richness, social presence, feedback quality, 
usefulness, and ease of use, there was no attempt to 
measure the possible impacts of different feedback 
channels (e.g., e-mail and v-mail) on learning outcomes. 

Second, this study only compared the quality of 
feedback provided by two Internet-based media (text- 
based e-mail and v-mail attachments sent via e-mail). It 
is impossible, based on the results from this study, to tell 
whether these Internet-based media provide a higher 
quality of feedback as compared to paper-based or face- 
to-face feedback. Future studies should be conducted to 
include both traditional and Internet-based media that 
can be used to provide feedback to students. 

Third, since all of the study subjects were graduate 
students majoring in computer information systems, 
generalizations to other populations should be made 
with caution. Similar studies should be conducted in 
other student populations as well as among employees 
who receive feedback from coworkers or supervisors. 

Fourth, in order to reduce the overall length of the 
questionnaire, perceived usefulness and perceived ease- 
of-use were both assessed by single item scales, which 
cannot be evaluated for reliability. Ideally, a multi-item 
scale should be used to assess each of those two 
measures. 

Finally, the finding that v-mail is perceived as having 
higher social presence than e-mail, but that the two 
media are not perceived as being significantly different 
in terms of media richness and media richness, suggests 
the need for further research into the subtle distinctions 
that may exist between these two constructs and the way 
in which they have been operationalized. 
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CONCLUSIONS 

This study has shown that both e-mail and v-mail can be 
used to provide timely, high quality feedback to 
students. Perceptions of social presence observed in this 
study were consistent with the predictions of SPT; v- 
mail was perceived to have higher social presence than 
e-mail. However, the two media were not perceived to 
be significantly different in terms of media richness, 
casting some doubt on the applicability of MRT in this 
context. These results suggest that while the two 
constructs are related, there may be important nuances 
of difference in their conceptualization and 
operationalization that merit further study. The findings 
also lend some support to the notion that perceived 
media richness may not result from an inherent property 
of the medium, per se, but may result from some 
combination of the medium, its properties, and the 
social context in which the medium is used. 
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APPENDIX 

CONSTRUCTS AND MEASURES 



Media Richness 

We are interested in your perceptions of the richness of this medium along the following dimensions. Information 
richness is defined as the potential information-carrying capacity of data. If the communication of an item of data, 
such as a wink, provides substantial new understanding, it would be considered rich. If the datum provides little 
understanding, it would be low in richness. This concept of richness is also believed to apply to the richness of 
information that can be carried on different communications media. This is known as media richness. For each 
dimension of media richness below, circle the number that best expresses your perception of [e-mail/attached 
V-mail file] as a communication medium. 

MRl. Giving and receiving timely feedback. 

Not at all Rich Extremely Rich 

1 2 3 4 5 



MR2. Transmitting a variety of different cues beyond the spoken message (non verbal cues). 

Not at all Rich Extremely Rich 

1 2 3 4 5 



MR3. Tailoring messages to your own or other personal characteristics. 
Not at all Rich 

1 2 3 



Extremely Rich 
5 



MR4. Using rich and varied language. 
Not at all Rich 

1 2 



Extremely Rich 
5 



Social Presence 

Different media are believed to have varying levels of social presence. The capacity to transmit information about 
facial expression, direction of looking, posture, dress, and non-verbal cues all contribute to social presence of a 
communications medium. For each dimension below, circle the number that best expresses your perception of 
the social presence of [e-mail/attached v-mail file] as a communication medium. 



SPl. Impersonal 








Personal 


1 


2 


3 


4 


5 


SP2. Cold 








Hot 


1 


2 


3 


4 


5 



Proceedings of the 17^ Annual Conference of the International Academy for Information Management 



109 



103 



SP3. 


Dehumanizing 








Humanizing 




1 


2 


3 


4 


5 


SP4. 


Insensitive 








Sensitive 




1 


2 


3 


4 


5 



Feedback Quality 

Please evaluate the quality of the feedback that you received on your exams along each of the following dimensions. 
The feedback I received on my exams was: 



FBQl Not Very Helpful 








Very Helpful 


1 


2 


3 


4 


5 


FBQ2 Not Very Useful 








Very Useful 


1 


2 


3 


4 


5 


FBQ3 A Little 








A Lot 


1 


2 


3 


4 


5 


FBQ4 Not Very Detailed 








Very Detailed 


1 


2 


3 


4 


5 



Usefulness 

Rate the usefulness of [e-mail/v-mail] as a medium for receiving feedback. 



Not at all Useful 








Extremely Useful 


1 


2 


3 


4 


5 


Ease of Use 











Rate the ease-of-use of [e-mail/v-mail] software as a medium for receiving feedback. 

Not at all Easy Extremely Easy 

1 2 3 4 5 
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ALTERNATIVE FOR UNDERGRADUATES? 
AN EXPERIMENT WITH THE STUDENTS IN 
GEORGIA, THE PROFESSOR IN AUSTRALIA 
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ABSTRACT 

This paper uses under-graduate students undertaking an on-line distance education class as the source for the 
evaluation of such classes. Although the sample size is small and there is some self-selection, it provides some 
preliminary answers to what such students see as the advantages and disadvantages of on-line education, whether 
these perceptions change with experience and to what extent such classes are effective. The results of the study 
reported in this paper suggest that some student students see on-line distance education in a very positive light, that 
these perceptions don *t change with experience and that the learning outcomes can be positive. 



INTRODUCTION 

Higher education, particularly in the business schools in 
the US, is facing some interesting demographics — the 
University of Phoenix (an on-line for-profit university) 
has become one of the largest markets for business text- 
book publishers, demonstrating its rapid growth in this 
field; the US professorate is aging and there will soon be 
many retirements — at a time when college enrollments 
will have to grow to deal with the “baby boomlet” 
(children of baby boomers currently approaching college 
age); and supply shortages in the information intensive 
fields are likely to be chronic in the professorate because 
of the pace of change and the financial rewards available 
in industry for qualified individuals. Not surprisingly, 
there is growing and potentially intense interest in 
alternative course staffing and delivery models among 
American academic administrators. 

The concept of enhancing education by the use of 
technology, and in particular the opportunity to move to 



different modes of delivery, continues to attract the 
attention of both practitioners and researchers. Impor- 
tantly, the research is moving into the mainstream 
professional literature, particularly the IS literature 
(Alavi. and Leidner, 2001, Picolli, Ahmad, and Ives, 
2001). The acceptance of such articles by these leading 
journals is indicative of the serious view of the research 
within the IS discipline. It is possible that the IS interest 
and focus results fi-om the acceptance of research into 
the use of the technology, or from the expectation that 
the computer communications technology involved 
presents less of a barrier to IS students and therefore is 
closer to the everyday activities of IS academics, or 
both. 

This paper reports the results of a study, conducted 
using as its base an on-line class of final year non-IS 
students (with an apparent propensity for on-line 
distance education) almost completely on-line. It 
attempts to deal with three main issues. 
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1 . The potential attractiveness (or unattractiveness) of 
on-line distance education to such students 

2. The role of student experience in on-line distance 
education 

3 . Satisfaction with, and effectiveness of, such courses 

The paper sets out the literature leading to the above 
issues and poses research questions along the lines of 
these issues, explains the methodology employed, 
outlines the data involved (and its limitations), presents 
a discussion of the results and concludes with general 
comments on this type of class and the limitations of the 
study. 

BACKGROUND 

There have been many calls in the literature for more 
research into particular areas of distance education. 
Alavi (1994) sought future studies, which wouldreduce 
the potential impact of student interactions by 
administering the study during different semesters or at 
different geographic locations (e.g., different campuses). 
Alavi, Wheeler and Bradley (1995) concluded that it 
was important to caitinue the inquiry into the 
effectiveness of collaborative teleleaming environments. 
They stated that with the declining cost and continued 
convergence of computing and communication 
technologies and the subsequent increase in prevalence 
of networked, multimedia computers, collaborative tele- 
leaming would be an increasingly viable educational 
alternative. They posed several specific questions for 
future research including: 

1. How do the learners respond to these new 
experiences? 

2. How effective are the new experiences? 

The call for further research was recently repeated 
(Alavi and Leidner, 2001). 

A model for the assessment of the possible take-up of 
online distance education has been proposed (Dick, 
Case, and Bums, 2001) — the model dealt with both the 
academic institution and student perspectives. In that 
paper, which reported the results of a preliminary study 
primarily aimed at assessing the validity of the model, 
it was suggested that on-line distance education would 
only suit some of the students, and only for some of the 
time. By and large the students (particularly the under- 



graduate students) in that study did not want on-line 
distance education due to a perception that there was 
more help available on campus, there were distractions 
at home, concentration periods would not be enhanced, 
there would be a diminished classroom experience and 
it would be costly to set up the home to facilitate on-line 
distance education. However the paper suggested that 
it might suit a small percentage of mainstream students 
who have the skills, resources and perhaps the need to 
not attend campus classes. 

While the model put forward by Dick et al. was heavily 
drawn from the telecommuting literature, (see 
particularly Belanger, 1999, Gray, Hodson and Gordon, 
1993, Mokhtarian and Salomon, 1996, and Tung and 
Turban 1996), it was noted in the papa" that there is 
considerable support for the issues from the distance 
education literature — accessibility, convenience, inter- 
national (or recognised) instructors and a “consumer 
orientation” (Alavi, Yoo and Vogel, 1997; Emmons, 
1999), and the ability to continue education or keep up 
to date while having only limited time available due to 
heavy work commitments (Jana, 1999; Boisvert, 2000). 
Likewise, many of the potential disadvantages — there is 
broad support for the notion that an educational 
programme is far more than a curriculum and that there 
are benefits from a “surround interaction” between the 
students, the instructor and the lectures. This rich 
variety of interaction is likely to be lost (Bertagnoli, 
2001). Others include not learning the skills to think on 
one’s feet, the absence of support and help, longer to 
develop a rapport between student and professor and 
cost issues related to tuition and technology (Emmons, 
1999). Attempts to measure satisfaction with distance 
education have been sporadic, other than the measure of 
enrolments and the growth in the number of institutions 
offering some form of distance education. One recent 
approach using the service industry as a base (Long, 
Tricker, Rangecroft and Gilroy, 2000) based the 
assessment largely on immediate application in the work 
place — not in an invalid measure, but perhaps only one 
of many. 

Recent papers have reported on success — Volery (200 1 ) 
attempted to relate success with technology, the 
academic, and the students’ previous use of the 
technology and Stallings (2002), has put forward the 
need for the institution to be well placed to measure the 
success of the on-line experiences. Piccoli et al. 
reported increased self-efficacy by students involved in 
a virtual learning environment, but dissatisfaction with 
the learning process. The academic institutions app^ear 
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ready to adopt a model where the computer is used as a 
conduit for teaching and learning (Marold, 2002) and 
the author points to several “successful” classes 
measured by student grade and conpared to more 
traditional classes (student satisfaction with the courses 
is not reported). 

One area commented on in much of the literature, but 
rarely studied, is that of experience with on-line distance 
education and whether such experience has an influence 
on perceptions of it (Alavi, 1994). While there is some 
work (for example Althaus, 1997) on the effects of 
computer literacy and familiarity with on-line distance 
education, it seems that many researchers feel that 
experience with on-line education affects perceptions 
and may affect the ability of students to undertake 
education using this form of delivery. 

The above literature gives rise to the following research 
questions: 

1 . What will a group of students who seem to have the 
propensity for on-line distance education see as the 
advantages and disadvantages and will they view 
these in a similar light to the large majority of 
students? 

2. Will a respondent’s perception of distance education 
change due to experience in this mode of delivery? 

3. Will under-graduate students be satisfied with such 
courses? 

METHODOLOGY 

A previously developed and validated questionnaire was 
used as the primary basis for the study. This question- 
naire was administered twice to all students in a class, 
once at he beginning and once at the end of the course. 
The returned questionnaires were not anonymous, 
enabling paired sample testing to be performed (a 
statistically stronger test than independent sample 
testing, which is normally performed when the 
individual’s responses at different points in time cannot 
be compared). A copy of the questionnaire maybe 
found as Appendix A. 

As stated above the survey had been validated 
previously (Dick, 2001) — initial reliability of the 
instrument in terms of stability was measured for that 
study by test-retest surveys and in terms of construct 



validity by Cronbachalpha scores todetermine internal- 
consistency reliability. 

Validity of the measurement instrument was assessed in 
terms of content validity, (specifically including face 
validity and sampling validity), empirical validity and 
construct validity. This methodology is in accordance 
with generally accepted procedure (Frankfort-Nachmias 
andNachmias, 1996). Specific procedures conducted to 
assess each of these for the current study were: 

• Face validity (a necessarily subjective assessment of 
the instruments’ appropriateness) was assessed and 
achieved by the researdier by using the previously 
validated study. 

• Sampling validity (whether a given peculation is 
adequately sampled by the measuring instrument) was 
provided by the distribution of the survey to all 
members of a class and by the researcher not 
following any particular bias in selection of the 
students to whom to distribute the survey. 

• Empirical validity was evaluated both by using 
measures contained within the survey to check the 
consistency of results and by comparing some parts of 
the survey document with another survey run 
independently (and anonymously) by the School 
management. 

• Construct validity was assessed by means of Cronbach 
alpha scores. 

External validity issues deserve special mentioa It will 
be noted in the next section of this paper (dealing with 
the data), that there are limitations to the generalisability 
of the conclusions that can be drawn from the results of 
the analysis of this data. In particular these relate to the 
particular characteristics surrounding the students in the 
class used for the study. Nevertheless it is believed that 
the study does provide a reliable basis for commenting 
on the on-line distance education issues as they relate to 
similar students and courses. 

In order to assess the effect of each of the independent 
variables on the preference for distance education, linear 
regression was performed on the data. Also a series of 
non-parametric paired t-tests (Wilcoxan) and was 
conducted to identify variations in the perceptions and 
preferences between the different groups of students in 
the data. 
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DATA 

The data for this study comprised respondents to a 
survey of on-line distance students undertaking a senior 
level Information Technology Management course for 
non-IS majors in the Business School at Georgia 
Southern University in the United States. The course is 
compulsory for Business School non-IS m^'ors and 
some 200 take the course each semester. The on-line 
class had 38 participants. All of these students were 
given the opportunity to complete the survey at the 
beginning and end of the course; an incentive was 
provided in the form of points for extra credit. In the 
event, 25 students completed the survey once and 15, 
twice. The response rates were therefore 56% and 40% 
respectively. Also, enhanced departmental anonymous 
course evaluations were used, partly to validate the 
survey findings and partly to address learning and 
management issues. 28 students completed these — a 
74% response rate. 

In the sample of 25, there were 12 males and 13 
females, only one student had any prior experience with 
any form of distance education, 23 were in the 21-25 age 
group with one in the 26-30 group and one in the 31-40 
group. 12 of the respondents indicated they were 
currently working, either full-time or part-time. 

The approach to the selection of the class and to the 
acceptance in that class deserves some explanation and 
consideration. It was decided to run the class in an on- 
line distance delivery mode after some students had 
already enrolled in what they assumed would be a 
traditional class. All of these students were contacted 
and were advised how it was proposed to run the course 
and offered the opportunity tojoin another class if they 
felt uncomfortable with the method being proposed. The 
students were also encouraged to discuss any concerns 
they might have wifli the instructor. At this stage there 
were 25 students enrolled. Few discussed the class with 
the instructor and none withdrew from the class. Other 
students were then offered the opportunity to enroll in 
the class, up to maximum of 40 (2 of whom though in 
the class lists, did not attend any classes and 
subsequently withdrew). 

A further relevant factor was that the class selected for 
on-line distance delivery was a class scheduled to meet 
at 5pm on Mondays and Wednesdays. This particular 
class was chosen for two reasons — it was intended to 
run “chat” sessions which would take place at the 
scheduled class times — 5pm in the relevant US time 



zone is early-mid morning the next day in Australia, 
where the instructor resided. The time was seen as 
convenient to both students and instructor. The second 
factor was that it was believed that a group of under- 
graduate students who enrolled in a 5pm class would be 
likely to be attracted to this mode of delivery due to the 
likelihood of work, family or other commitments. 

Finally a word about the class, as the conduct of the 
class may also be relevant to the perceptions of on-line 
distance education. The instructor visited Georgia and 
conducted one face-to-face class in week one of the 
semester and outlined how the rest of the semester 
would unfold. All students were encouragedto visit with 
the instructor on an individual basis, particularly anyone 
who had some concerns about the use of the technology. 
About 30% of the students availed themselves of this 
opportunity. The class was conducted primarily via 
WebCT and the students had a textbook, a PowerPoint 
presentation, study notes and an audio file for each 
week’s work. Assessment and deliverables were based 
on each week the students partaking in a “chat” session, 
a contribution to a “discussion board”, an assignment, 
and an on-line quiz on that week’s prescribed textbook 
chapter. With the exception of the “chat” session, all of 
these were asynchronous. Other assessment conponents 
included on-line quizzes and a supervised exam 
conducted on WebCT. 

There are obvious limitations to the findings from the 
data used in this study. These are discussed in more 
detail at the conclusion of this paper, however it should 
be noted (as outlined above) that prior research had 
suggested that on-line distance education would be best 
suited to only some students, for only some of the time. 
This was an attempt to determine whether an under- 
graduate course could be successfully run using students 
who prima facie seemed that they would be receptive to 
on-line distance education and limiting their exposure to 
just the one course. 

RESULTS AND DISCUSSION 
Advantages 

Over 90% of the respondents agreed or strongly agreed 
that being able to choose the time to study and work on 
assignments was important to them. Other statements 
with which there was high agreement were that distance 
education would reduce travel and commuting costs, 
allow better management of work commitments, enable 
taking care of dependents, provide flexibility, and that 
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fewer distractions in the home meant it allowed them to 
be productive distance students. 

While the overall preference for distance education was 
neutral among these respondents, regression analysis, 
running the potential advantages against the preference 
for distance education, indicated that being able to 
choose the time for study, managing work commitments 
and being able to work at home (few distractions) were 
the predominant variables in that preference — = .517. 

The respondents were also asked to rate what they saw 
as the most important (and least important) potential 
advantages of distance education. Almost all (95%) 
respondents rated being able to choose time to study and 
work on assignments as one of the three most important 
and 64% chose being better able to manage work 
commitments. Only 15% rated the latter as unimpor- 
tant, none rated the former as unimportant. 

The picture emerging here indicates that for this group 
of respondents, distance education provided a sought 
after alternative to their normal mode of study. 

Disadvantages 

When considering the potential disadvantages of on-line 
distance education, most students in this class had no 
real concerns. They did not see it as difficult to study at 
home due to less help, motivational problems, etc., they 
were not concerned about not being able to avail 
themselves of the resources on campus and did not see 
obtaining the necessary resources as difficult. They also 
agreed contact with the instructor was not difficult. 

Stepwise regression, running the potential disad- 
vantageous variable against the preference for distance 
education indicated that being unconcerned about 
missing out on the available resources was the most 
prevalent factor (R^ = .149). The students also indicated 
that they saw the most importantpotential disadvantages 
as the diminished classroom experience and missing out 
on the professional interaction with other students. 

Taking the advantages and disadvantages together, it 
seems that the driving factors in the preference for 
distance education are (R^ = .673.): 

• Being able to choose the time and place for study, 

• Not being concerned about unavailable resources, 



• Being better able to manage work commitments, and 

• Fewer distractions at home allowed the student to be 
productive. 

The limitations of the study in terms of the ability to 
generalise to the student population as a whole are 
clearly evident here. These students were mostly 
seniors, and many lived on campus or close to campus. 
Accordingly it might be expected that they would 
possess a level of maturity and ability commensurate 
with that required for such a course and in any case they 
had to come to campus (and its available resources) for 
other courses. Nevertheless an objective in conducting 
this study was to determine whatfactors might influence 
this particular type of student. These factors differ 
markedly from the factors as reported (Dick etal., 2001) 
for the wider student body. 

Do Perceptions Change with Experience? 

A series of Wilcoxon Signed Ranks tests (the non- 
parametric tests were used due to the small sample size 
and the non-normal distribution of the data) revealed 
virtually no differences in the perceptions of the 
students in respect of distance education over the course 
of the class. Only in one item was there a significant 
variation (at the .05 level) that of a potential reduction 
in living costs due to being able to live at home — the 
respondents were less likely to agree with this at the 
conclusion of the course than they were at the 
beginning. This is understandable, the respondents 
attended only one class via distance and would have 
continued with their normal living arrangements. It is 
also worthy of note that only two of the respondents had 
any previous experience with distance delivery and a 
series of independent sample t-tests, both parametric and 
non-parametric revealed no significant variations. 

The respondents were asked to indicate their degree of 
agreement with the statement “I prefer distance 
education courses over traditional classroom based 
courses.” On a scale of 1-5, where 1 indicated strong 
agreement and 5 indicated strong disagreement, there 
was a minor movementin student perceptions during the 
course. The mean moved from 2.8 (slightly on the 
“agree” side of neutral) to 3.3 (slightly on the “disagree” 
side of neutral). The movement was not statistically 
significant — p = .084, although certainly could be 
considered as trending in that direction. 
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Nevertheless, it seems reasonable to conclude that on 
the basis of this data, that after taking a distance 
education class, perceptions of distance education will 
remain the same as at the beginning. There is a potential 
qualifying factor here, due to a limitation in the 
data — by and large the students were very happy with 
this particular distance education class. Had this not 
been so, it is possible that some of the perceptions 
would have changed. 

Satisfaction and Effectiveness 

In an attempt to address these issues, a number of 
factors were considered. In addition, a separate set of 
data is included in the analysis — that collected 
anonymously and independently of the instructor by the 
Department at the time of the final examination. This 
comprised a standard course evaluation questionnaire, 4 
extra questions specifically related to satisfaction issues 
and 2 open-ended questions that gave the students the 
opportunity to make general comments. The results of 
the four open-ended questions are given in Figure 1 
below, details of the actual questions and the open- 
ended questions are given in Appendix B. 

FIGURE 1 

COURSE PERCEPTIONS 
(INDEPENDENT EVALUATIONS) 




mick? 



□ Yef 
mtto 



Factor 



There is considerable similarity between the results for 
the first two factors in Figure 1 and those in Figure 
2 — part 6 of the survey form (Appendix A), which was 
not anonymous. It would appear from this data that the 
students were less enthusiastic about the course on the 
forms where the instructor could identify them and more 
enthusiastic on the anonymous forms. Among other 
things, this finding gives some weight to the validity of 
samples from students where often the respondent can 
be at least partially identified. That aside, it seems there 
was widespread satisfaction with the course, at least as 
a learning experience. 



FIGURE 2 

COURSE PERCEPTIONS 
(SURVEY RESPONSES) 
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In terms of learning outcomes, there were a number of 
relevant factors in the department evaluations — these 
invited the student to compare the course to other 
similar courses that they had taken. The results of these 
evaluations are given in Figure 3, but in general terms it 
seems that for most students the learning outcomes were 
about the same as for other courses, with the 
qualification that around 30% of students felt that they 
learnt more, it was more intellectually challenging and 
that it was more difficult. Against this, a small 
percentage (around 5%) fek that they learnt less, were 
less challenged and that it was less difficult 

The results shown in Figure 3 were compared with the 
results for other sections of the same course run in that 
and the subsequent semester. No significant differences 
were noted. That is, all students taking this course felt 
the same way about it in terms of effcat, learning, 
challenge and degree of difficulty, regardless of the 
mode of delivery. There was perhaps an indication in 
the data that the online group felt slightly more 
challenged (a mean of 3.4 to 3.1 on a 5 point scale) 
however this is not considered to be significant given 
the limitations of data numbers and the question 
wording. 

While not central to the learning issue, the result from 
the independent evaluation rating the course is given 
below (Figure 4) — it does indicate general satisfaction 
but may also be relevant when considering the general 
findings outlined in this section. It is also worthy of note 
that in response to the open-ended questions in 
Appendix B, most commented favourably on their 
interaction with the instructor and there were veiy few 
suggestions put forward for improvements. 
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FIGURE 3 

LEARNING OUTCOMES 
(INDEPENDENT EVALUATIONS) 




HOW WOULD YOU RATE THIS COURSE? 
(INDEPENDENT EVALUATIONS) 




CONCLUSIONS, LIMITATIONS, AND 
FURTHER RESEARCH 

It seems reasonable to conclude that for some under- 
graduate students on-line distance education is a viable, 
attractive alternative. In this case, about 20% of the 
students taking the course were enrolled in the on-line 
class. Such students see the advantages and disadvant- 
ages of this mode of delivery in a different light to the 
wider student body. It also seems that such students are 
likely to be more senior, comfortable with the 
technology and have work or home commitments that 
make regular attendance at classes difficult or 
undesirable. On the basis of this study, it is reasonable 
to state that these perceptions hold up, for at least some 
time, with experience in such courses. 



There is also evidence from this study that such courses 
can be satisfying and effective, from the student’s 
perspective. Indeed this study provides evidence (self 
management issues and technological frustrations aside) 
that academically, the learning e?q)erience was very 
similar to that achieved by those in the more traditional 
mode and to that achieved by these students when 
undertaking traditional classes, perhaps enhancing its 
attractiveness to this particular group of students. 

There are clear limitations to this study, as mentioned in 
several places above. These centre around the self- 
selection and small size of the sample, however given 
that one of the objectives was to assess such a group of 
students, the self-selection issue has less impact than 
might otherwise be the case. Care obviously needs to be 
taken in extrapolation of these results to the wider 
student body. A larger sample would improve the 
reliability of the results, too. 

One measure not covered in this study was a comparison 
of student results with other students taking the course. 
It was felt that different instructors, different students 
and the small sample size would all work against 
meaningful findings, however it is suggested that in a 
further study this would be a useful comparison. 
Ideally, the same instructor would take more than one 
class, some (but not all) in on-line distance deliveiy 
mode. Results could then be compared across classes 
and with the student’s results in other subjects. 

Another area for further work relates to the role of the 
instructor — the effort involved, both in setting up the 
course/s, the time and work involved in dealing with 
queries and marking by distance, the administration of 
quizzes and examinations and the instructors’ percep- 
tions of the success of the course. This work is most 
likely to involve a series of case studies, as there are 
many different models of on-line distance education in 
use, which would cloud findings and make analysis 
difficult. 

This study has taken a small step along the road of 
student responses to, and satisfaction with, such 
courses — there is still a long way to go. 
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APPENDIX A 

DISTANCE EDUCATION SURVEY 



I . Please indicate the extent to which you agree that the following potential advantages of distance education apply to you: 

Strongly Strongly 







agree 


-> -> -> 


-> -> 


->-> -> 


-disagrei 


a 


Distance education allows me to reduce travel and commuting costs. 


□ 1 


□2 


□ 3 


□4 


□ 5 


b 


Distance education allows a reduction in living costs due my ability 
to live at home, not on campus. 


□ 1 


□ 2 


□ 3 


□4 


□ 5 


c 


Distance from home would make class attendance on campus 
impossible. 


□ 1 


□ 2 


□ 3 


□4 


□ 5 


d 


I am better able to manage work commitments by not being required 
to attend class on campus. 


□ 1 


□2 


□ 3 


□4 


□ 5 


e 


Distance education enables me to complete class work if disabled or 
taking care of dependents. 


□ 1 


□2 


□ 3 


□4 


□ 5 


f 


I find the campus environment intimidating or undesirable. 


□ 1 


□ 2 


□ 3 


□4 


□ 5 


g 


Personal reasons such as family, flexibility make distance education 
attractive to me. 


□ 1 


□ 2 


□3 


□4 


□ 5 


h 


Fewer distractions for me at home allows me to be a productive 
distance education student. 


□ 1 


□ 2 


□ 3 


□4 


□ 5 


i 


Being able to choose the time to study and work on assignments is 
important to me. 


□ 1 


□2 


□ 3 


□4 


□ 5 



PLEASE INDICATE UP TO 3 OF THE ABOVE THAT YOU SEE AS THE MOST AND LEAST IMPORTANT: 

Most important Least important 



2. Please indicate the extent to which you agree that the following potential disadvantages of distance education apply to 
you: 

Strongly Strongly 







agree 


-> -> -> 


-> -> 


->-> -> 


-disagree 


a 


I would find it more difficult to study at home due to less help, 
motivational problems, increased family conflicts, distractions. 


□ 1 


□2 


□ 3 


□4 


□ 5 


b 


I feel that there is better help for me available on campus. 


□ 1 


□2 


□ 3 


□4 


□ 5 


c 


I would miss out on benefits available on campus — resources, 
possible employment etc. 


□ 1 


□ 2 


□ 3 


□4 


□ 5 


d 


The cost of procuring the necessary distance education equipment 
for my home would be expensive for me. 


□ 1 


□ 2 


□ 3 


□4 


□ 5 


e 


I would miss out on the extra-curricular activities available on 


□ 1 


□ 2 


□ 3 


□4 


□ 5 




campus. 




f 


Missing out on the professional interaction with one’s fellow 
students would be a concern to me. 


□ 1 


□2 


□ 3 


□4 


□ 5 


g 


There would be a diminished classroom experience — less 
discussion, interaction with professors. 


□ 1 


□2 


□ 3 


□4 


□ 5 


h 


I find it very difficult to contact the instructor. 


□ 1 


□ 2 


□ 3 


□4 


□ 5 



PLEASE INDICATE UP TO 3 OF THE ABOVE THAT YOU SEE AS THE MOST AND LEAST IMPORTANT: 

Most important Least important 
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3. Thinking about your academic work, please indicate your degree of agreement with the following statements as they 
typically concern course related tasks you have to complete (study, assignment work, exam preparation etc.): 



a The final product of the tasks that I am assigned typically involves 
the comp letion of many different comp onents. 
b My tasks often require me to work with fellow students, 
c Sometimes the task deliverables change over the duration of the 
assignment (e.g. the instructor adds or deletes one or more 
components). 

d Tasks I am given are not always clear and may be interpreted in 
different ways. 

e I am often uncertain about what to do to complete the final product, 
f The tasks I am assigned are often dependent on at least one other 
student completing his work first. 

g The task deliverables are clear, but can be accomplished in a number 
of ways. 

h I would typically rather work on my own, than with other students, 
i The tasks I am assigned require minimal resources (e.g. software, 
library, etc.). 

j The tasks I am assigned allow me to work at my own pace, 
k Assigned tasks require long periodsof concentrated attention. 

1 Distance education enables me to concentrate on course related tasks 
for long period s. 

m “Due dates” for tasks assigned are clearly stated, 
n There is a need for a considerable degree of communication with my 
fellow students. 

0 There is a need for a considerable degree of communication with 
academic staff. 



Strongly Strongly 



agree 


A 

A 

A 


-> -> 


->-> -> 


-disagrei 


□ 1 


□ 2 


□ 3 


□4 


□ 5 


□ 1 


□ 2 


□ 3 


□4 


□ 5 


□ 1 


□2 


□ 3 


□4 


□ 5 


□ 1 


□ 2 


□3 


□4 


□ 5 


□ 1 


□ 2 


□ 3 


□4 


□ 5 


□ 1 


□ 2 


□ 3 


□4 


□ 5 


□ 1 


□2 


□ 3 


□4 


□ 5 


□ 1 


□ 2 


□3 


□4 


□ 5 


□ 1 


□ 2 


□ 3 


□4 


□ 5 


□ 1 


□ 2 


□ 3 


□4 


□ 5 


□ 1 


□ 2 


□ 3 


□4 


□ 5 


□ 1 


□ 2 


□3 


□4 


□ 5 


□ 1 


□ 2 


□ 3 


□4 


□ 5 


□ 1 


□2 


□ 3 


□4 


□ 5 


□ 1 


□ 2 


□ 3 


□4 


□ 5 



4. Thinking about your ability to study via distance education, please indicate your degree of agreement with the following 
statements: 



Strongly Strongly 







agree 


-> -> -> 


-> -> 


->-> disagree 


a 


I am capable of making good decisions about the tasks I am assigned. 


□ 1 


□ 2 


□3 


□4 


□ 5 


b 


I know where to get the relevant information I need to complete the 
assigned tasks. 


□ 1 


□ 2 


□ 3 


□4 


□ 5 


c 


I have no difficulty determining when I should seek advice. 


□ 1 


□ 2 


□3 


□ 4 


□ 5 


d 


I am good at selfmanagement, possessing the motivation, time 
management, etc. that is needed to deliver quality work on time. 


□ 1 


□ 2 


□ 3 


□4 


□ 5 


e 


I have the computing and communications resources I need to be an 
effective distance education student. 


□ 1 


□ 2 


□ 3 


□4 


□ 5 



5. Please indicate the extent of your agreement with the following statements: 







Strongly 






Strongly 






agree -> -> -> 


-> -> 




disagree 


a 


Distance education is of lesser quality than traditional class-room- 
based campus education. 


□ 1 D2 


□ 3 


□ 4 


□ 5 


b 


I only participate in distance education because I can’t attend 
campus classes. 


□ 1 D2 


□3 


□4 


□ 5 


c 


The tasks associated with my course are suitable for the distance 
education environment. 


□ 

□ 

N.) 


□ 3 


□4 


□ 5 


d 


I should not have to pay as much for distance education as for 
traditional campus based education. 


□ 1 m 


□ 3 


□4 


□ 5 
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Strongly 








Strongly 






agree -> 


-> -> 


-> -> 




disagree 


e 


I would encourage most professionals to participate in distance 
education. 


□ 1 


□ 2 


□ 3 


□4 


□ 5 


f 


Instructors should not assign the same tasks to distance based 
students as they assign to campus based students. 


□ 1 


□ 2 


□ 3 


□4 


□ 5 


g 


Distance education courses are designed with the distance student in 
mind. 


□ 1 


□ 2 


□ 3 


□4 


□ 5 


h 


I believe I have the skills and ability to be a successful distance 
educatio n student. 


□ 1 


□ 2 


□ 3 


□4 


□ 5 


i 


Distance education is attractive to Universities because it provides 
additional revenue without the need for additional resources. 


□ 1 


□ 2 


□ 3 


□4 


□ 5 


j 


I prefer distance education courses over traditional classroom based 


□ 1 


□ 2 


□ 3 


□4 


□ 5 




courses. 


k 


Distance education is an acceptable instructional delivery system, 
but it falls short of the traditional classroom experience. 


□ 1 


□ 2 


□ 3 


□4 


□ 5 



Thinking specifically about the MGNT 4130 course you are now doing: 
a This course has been a wonderful experience for me. 


□ 1 


□ 2 


□ 3 


□4 


□ 5 


b 


I wish I could do/have done more courses in this way. 


□ 1 


□ 2 


□ 3 


□4 


□ 5 


c 


I would have liked more help with the “technology.” 


□ 1 


□ 2 


□ 3 


□4 


□ 5 



The best things about this course were: 



The following areas need improvement: 



7. Please provide the following general information about yourself: 

1. Are you : ^ Male □ Female 

2. In which age group are you? 

□ 20 or younger □21-25 □ 26 - 30 □31-40 

□ 41-50 □ 51-60 □ over 60 

3. How many people in your household (including yourself) fall into each of the following age groups: 

Under 2 years old 2-5 years old 6 - 15 years old 

16-20 years old 21-65 years old over 65 years old 

4. How many courses have you taken by distance education (before this one)? q □! 1-3 ^2 > 3 □S 
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APPENDIX B 



1. Did you enjoy your distance learning experience? 

2. Would you take another course that was taughtby distance learning? 

3. Would you recommend a similar distance learning course to a friend? 

4. Do you feel that you learned as much as if you were in a traditional classroom? 

1. What did you like best about this instructor/course? 

2. How could this instructor/course be improved? 
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ABSTRACT 

This research looks at the pedagogical implications and the characteristics of learning in three different instructional 
environments. Not only is the set of circumstances of the instruction delivery important, but perhaps more important 
is the content organization and information provision. Each of the three instructional environments is examined 
under nine instructional events and how they impact the learning process. The evaluation scheme and example 
presented in this paper are intended to represent a framework for evaluation in selecting instructional materials and 
developing an approach to teaching. Comprehensive evaluations of a learning environment should focus on the 
content organization and information provision, and only subsequently on the methods of instructional delivery. 



INTRODUCTION 

Computer-mediated instruction, either in the form of 
interactive Web-based instruction, electronic forum, 
synchronous discussion rooms, or simply self-paced 
interactive multimedia has become an important aspect 
of today’s pedagogy. Through the spread of Web-based 
courseware packages, Universities have adopted user- 
friendly multiple-media based tools to create a more 
stimulating learning environment. Face-to-face instruc- 
tion is supplemented with different forms of computer- 
mediated instruction in dissimilar disciplines and 
content areas. 

Wirth and Amos (1996) identify different delivery 
methods of instruction (either stand-alone or supple- 
mental to traditional instruction) and then rank these 
methods on an intensiveness scale. This scale is the set 
of circumstances of the instruction delivery. Although 
the mode of delivery is an important element of a 
comprehensive review of the effectiveness of the 
learning experience, there are other factors equally (or 
even more) important. Evaluation of a learning 



environment should be more focused on the content 
organization and information provision, rather than the 
methods of instructional delivery. Gagne’s model 
(1992) details nine instructional events relevant to 
effective learning. This research uses Gagne’s model 
combined with several other program criteria to evaluate 
how the content organization of different delivery 
methods may stimulate learning conditions and 
therefore, learning. 

This study reviews specific learning conditions and 
presents a framework for evaluating the effectiveness of 
the elements of the learning environment used to engage 
students and increase performance. The adoption of a 
model to evaluate the content and delivery modes of 
instruction is a pre-requisite to understanding which 
elements of the students’ learning experience are to be 
supplemented. Although the course studied in this 
research is introductory Project Management, the same 
reasoning may be applied to any subject area. Because 
of the availability and use of different instructional 
environments based on the same subject and developed 
by one instructor, this course was selected for this study. 
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Instructors and faculty can take the framework into 
account when evaluating the different components 
(synchronous or asynchronous) of their course delivery. 
Those areas in the instructional environment that best 
support learning and retention can be strengthened, 
thereby providing a better educational experience for 
students. 

Initially, the paper presents an important evaluation 
framework that has been used in the literature to 
compare different methods of instruction (Wirth and 
Amos, 1996). Then a more comprehensive model 
(Gagne, 1992), focused primarily on cognitive 
processes, is introduced. This model is tiien applied to 
compare instructional delivery modes inan introductory 
Project Management course. By contrasting results 
obtained using Gagne’s framework with conclusions 
from Wirth and Amos (1996), the authors explain the 
importance of focusing first on the evaluation based on 
criteria related to content organization, and only 
secondarily, on delivery types^media. Contrary to some 
authors (McLuhan and Fiore, 1967), the medium is not 
the message. The emphasis should remain on assessing 
content, and the comprehensiveness of its presentation. 

EVALUATION OF DELIVERY 
METHODS OF INSTRUCTION 

A review of project management instruction programs 
(Wirth and Amos, 1996) identifies over 20 distinct 
delivery methods of instruction. Wirth and Amos rank 
these programs on an intensiveness scale, ^ which they 
define as the set of circumstances of the instruction 
delivery. Interactivity^ is a key element of learning 
(and, therefore, a key element of the intensiveness 
scale), with higher interactivity generally displaying 
positive impact on learners’ performance and attitudes. 

Wirth and Amos evaluate the intensiveness of different 
Project Management instructional programs (Figure 1) 
and find that the most effective are interactive seminars 
(formal seminars with company specific emphasis, with 
case study discussion; generally 2-day duration), and 
experiential seminars (seminars involving discussion of 
training materials; topical emphasis on reviews and 
evaluation of project cycles; with a typical duration of 3 
hours). 

Classroom instruction (lecture/discussion; individual 
project reports and case studies; topical emphasis on 
project management competencies, and less on human 
skills; 16-week duration) ranks in the 6‘*' position; closely 



followed by multimedia training programs (defined in as 
many as ten-30-minutes TV-series/ textbook, with 
exercises consisting in computer-mediated multiple 
choice tests, topical emphasis on generic project 
management) in the place. Computer-based training 
(CBT — the automated version of programmed instruction 
by an interactive workbook for in-depth exploration of 
general project management topics) ranks in the lowest 
position (of the three environments evaluated) on the 
intensiveness scale (10^^). 

Overall, results of Wirth and Amos (1996) analysis 
indicate that face-to-face instruction is most effective. 
The use of multiple media (such as audio-tapes, visuals, 
text and other media — not necessarily synchronized) is 
the second most important. Last, the use of computer- 
mediated instruction (CM) is the least preferred as it 
entails lower level of intensiveness of the learning 
experience. Although these results may represent an 
important picture of the effectiveness of instructional 
programs, it is crucial to evaluate instructional delivery 
on several other parameters that take into account 
students’ cognitive processes engagemait. The authors 
of this paper argue that rather than focusing primarily on 
the delivery medium, the evaluation of learning needs to 
be based on the organization and clarity of the ^content ' 
of instruction, that is, its ability to initiate the conditions 
of learning (recqytion, reinforcement, retrieval, etc.) to 
stimulate long-term recall and retention. Therefore, the 
authors propose an evaluation mainly based on Gagne’s 
learning conditions and define the elements that should 
be evaluated to apply Gagne to different instructional 
delivery environments (i.e., face-to-face, textbooks, 
CBT) .The evaluation model proposed should focus more 
on the content organization and information provision, 
rather than the methods of instructional delivery. 

CONDITIONS OF LEARNING 

Gagne (1992) identifies two types of conditions of 
learning. The internal conditions include the 
intellectual skills, cognitive strategies, attitudes, and 
other circumstances that the learners bring to the 
learning environment. The external conditions are the 
instructional events that influence the individual internal 
state (learning, thinking, and remembering) and contri- 
bute to knowledge creation. These external events of 
instruction correspond to specific cognitive processes 
that influence individual interactions with the 
information. Effective learning occurs when combining 
individual cognitive events with instructional strategies 
that maximize the impact of the external conditions on 
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FIGURE 1 

EVALUATION OF PROJECT MANAGEMENT INSTRUCTIONAL PROGRAMS INTENSIVENESS 



Program 


Location 


Instructor 

proximity 


Instruction 

period 


Interaction 

frequency 


Interaction 

duration 


Interactive 

type 


Total 

Score 


Delivery 

Tech- 

nology 


Rank 

order 


Interactive seminar 


4 


5 


5 


5 


4 


5 


28 


FF 


I 


Experiential seminar 


3 


5 


5 


5 


4 


5 


27 


FF 


II 


Symposium seminar 


2 


5 


5 


5 


5 


5 


27 


FF 


II 


Professional seminar 


2 


5 


5 


5 


5 


5 


27 


FF 


II 


Practitioner seminar 


2 


5 


5 


5 


5 


5 


27 


FF 


II 


Project team 


5 


5 


3 


4 


4 


5 


26 


FF 


III 


education 


In-housc/OJT seminar 


4 


5 


4 


5 


3 


5 


26 


FF 


III 


In-house hybrid 


4 


5 


4 


4 


3 


5 


25 


MM 


IV 


course 


Audio-graphic 


4 


4 


4 


4 


3 


5 


24 


MM 


V 


systems 


Classroom instruction 


3 


5 


2 


4 


3 


5 


22 


FF 


VI 


Teleconferencing 


4 


4 


4 


3 


3 


5 


22 


MM 


VI 


Multimedia training 


1 


4 


4 


4 


3 


3 


19 


MM 


VII 


Electronic mail 


1 


3 


4 


3 


2 


5 


18 


MM 


VIII 


Correspondence 


1 


4 


1 


2 


2 


5 


15 


MM 


IX 


course 


Expert-system self- 


1 


2 


1 


2 


2 


4 


12 


CM 


X 


training 


Dynamic/intcractivc 


1 


2 


1 


2 


2 


4 


12 


CM 


X 


computer simulation 


Computer-based 


1 


2 


1 


2 


2 


4 


12 


CM 


X 


(CBT) 


Computer simulation 


1 


2 


1 


2 


2 


4 


12 


CM 


X 


games 


Audio-visual tape 


1 


2 


1 


2 


2 


4 


11 


MM 


XI 


Audio-tape 


1 


2 


1 


2 


2 


3 


11 


MM 


XI 



Legend: FF= Face-to-Face MM= Multiple media (not necessarily synchronized) CM= Computer-mediated 



Scale: I (low)-> 3 (medium) -> 5 (high) 

Source: Wirthand Amos (1996) 



knowledge creation, storage and retrieval. Gagne’s 
model (Figure 2) is an appropriate instrument to 
evaluate how instructional materials create the external 
conditions that promote students’ learning. The internal 
cognitive processes are activated by the nine 
instructional events, which represent the fundamental 
blocks of a program (external condition) relevant to 
effective learning. 

The evaluation framework presented in this paper uses 
a combination of techniques (descriptive general 
approach and specific software checklist) derived from 
the integration of the ‘TECC and California Library 
Media Consortium for Classroom Evaluation of 
Microcomputer Courseware’ and the ‘National Council 
of Teachers of Mathematics (NCTM) Software 



Evaluation Checklist’ (Appendix A). The TECC 
evaluation guidelines are used mainly to guide the 
descriptive evaluation. Gagne’s model is finally added 
to the model to take into account how the program is 
capable of stimulating the learner’s internal conditions. 

In the remainder of the paper, sample evaluations are 
presented with reference to teaching introductory Project 
Management through different instructional deliveries 
(CBT with interactive multimedia, textbooks and face- 
to-face). Although all program evaluation criteria are 
assessed for each instructional delivery environment, the 
main focus of the evaluation is on the assessment of the 
‘effective use of the learning events’ as identified in 
Gagne (1992). 
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FIGURE 2 

CONNECTION OF INSTRUCTIONAL EVENTS WITH LEARNING 



Gagne's (1992) Instructional Event 


Learning Process 


1 . Gaining attention 


Reception of stimulus and activating of sensory registers 
(Reception) 


2. Informing the learner of the lesson objectives and 
activating motivation 


Activating of executive control (Expectancy) 


3. Simulating recall of prior learning 


Retrieval o f inform aiton to working memory (Retrieval) 


4. Presenting the stimulus material 


Emphasizing features for selective perception (Selective 
Perception) 


5. Providing learning guidance 


Semantic encoding; cues for retrieval (Semantic Encoding) 


6. Eliciting performance 


Activating response (Responding) 


7. Providing feedback 


Activating retrieval (Reinforcement) 


8. Assessing performance 


Providing cues and strategies for retrieval (Retrieval) 


9. Enhancing retention and learning transfer 


Repeat in subsequent lessons; provide additional diverse 
examples (Generalization) 



Adapted from Shambaugh and Magliaro (1997, p. 197) 

The criteria of evaluation of program cognitive 
effectiveness (and the questions they answer) are 
summarized in Table 1. 

A SAMPLE EVALUATION 

An example of how the evaluation questions are applied 
to assess the completeness of different Project 
Management teaching materials is presented In this 
specific example, the purpose is to generally assess the 
overall strength of the instructional medium and present 
an overview of relevant classification criteria to quickly 
identify needed intervention areas to supplement the 
programs. The content being evaluated refers to two 
introductory Project Management book titles (Frame, 
1994; 1995). Particularly, the textbooks, the face-to- 
face instruction based on the books, and the multimedia 
version of the textbooks are benchmarked. The interval 
scale of the sample evaluation uses Harvey-ball format 
(from very low to very high). Other scales may be 
substituted (i.e., Likert). 

Evaluation of Interactive Multimedia 

‘Project Management in Organizations (PMO) CD- 
ROM’ is the multimedia version of Frame, 1994 and 



1995. The CD-ROM closely follows the books’ 
organization: learning units mirror chapter titles and 
structure. The application is designed to teach the 
fundamentals of project management. Audiences vary 
from project managers, project team members, 
executives, or other stakeholders interested in learning 
project management. [ORGANIZATION] 

The CD-ROM provides a stand-alone tool for learning 
project management introductory topics. It is used as a 
support tool for distance learning, and requires little 
synchronous interaction. Users are expected to work 
with the software independently, at their own pace. The 
software includes practice tests and feedback features to 
self-assess the level of competency reached by using the 
application. [MANAGEMENT] 

The strength of the software is in the use of video and 
audio to capture attention, provide simulations of real- 
world scenarios, and give feedback. Simulations 
reproduce office-like settings (Figure 3) (i.e., telephone 
on office desk, with blinking light indicating a new 
message, and presenting instructions on how to 
proceed). Navigational aids are available for the user in 
each step. Contextual instructions appear consistently 
at the bottom of each screen. The notepad and 
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TABLE 1 

PROGRAM EVALUATION CRITERIA 



• Organization (TECC) [ORGANIZATION] 

• Content 

• Is content relevant, complete, and well developed? 

• Objectives 

• Are the objectives clearly stated and achievable throughout the program? 

• Users interaction with the program 

• Are the means of interaction between the learner and the content user-friendly? 

• Management system (including scoring or performance) (TECC) [MANAGEMENT] 

• How well is the achievement of the learning objectives managed and measured? 

• Special strengths [STRENGTHS] 

• Are there relevant characteristics/features that make the instructional program particularly strong? 

• User orientation: instructor’s point of view (NCTM) [INSTRUCTOR] 

• Is the role of theinstructor marginal in providing learners' orientation with the content? Is the instructor freed 
from the need to intervene? 

• User orientation: user’s point of view (NCTM) [USER] 

• Is the learner able to independently access the content? Are directions self-explanatory? 

• Motivation and instructional style (NCTM) [MOTIVATION] 

•Are there sufficient elements and reinforcement that engage the learners and maintain their interest in the 
lesson content? Is the instructional style engaging? 

• Social Characteristics (NCTM) [SOCIAL] 

• Does the instructional approach provide cooperation or competition? Does it promote teamwork? Is the 
content appropriate for all audiences? 

• Effective use of learning events (Gagne) [LEARNING] 

• Are the majority of Gagne's learning events achieved? Is the instructional style geared towards leveraging 
learners ' cognitive skills? 



FIGURE 3 

CD-ROM PRACTICE MENU 
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calculator shortcuts offer easy access to supporting 
tools. [STRENGTHS] 

The program frees the instructor from the need to 
intervene; therefore, it is particularly useful for self- 
paced instruction. However, the CD-ROM does not 
allow for flexibility of instruction as related to skill 
levels. There is only one level for all users, which 
makes the software particularly useful for users with low 
prior knowledge of the topic, however, it may become 
tedious for those with some level of competency. On the 
other criterion established by the “National Council of 
Teachers of Mathematics’ software evaluation 
checklist” there is an almost perfect freedom from need 
to intervene or to assist and the application can be used 
stand-alone (Appendix A). [INSTRUCTOR] 

The software provides high quality directions (clarity), 
quality output (content and tone), pleasant screen 
design, freedom from the need of external information, 
and freedom from disruption by system errors. The 
software requires different forms of input. The user is 
often asked to provide a wide range of responses at 
different levels of interactivity: from typing full 
answers, to drag and drop, and to performing complex 
calculations. [USER] 

The software uses graphics, videos, and audio clips that 
increase motivation. The simulations and case studies 
are organized as games to foster interaction, thinking 
and retention through reinforcement. The student is 
continuously engaged in new activities. Occasionally, 
some activities do not allow unlimited student control; 
the user is forced to complete each action, before being 
able to proceed. The problem-solving activities are very 
structured and software-led (as opposed to user-led). 
[MOTIVATION] 

Because of its stand-alone nature (one user), the 
software does not allow competition or cooperation. 
Users can use the software while working in teams, but 
no formal mechanism is included in the design to 
facilitate team-usage. No specific navigational 
functions provide for more direct interaction between 
the narrator and the user. Therefore, no humanization of 
the computer is obtained. However, the software does 
provide a summary of students’ performance (in the 
testing areas) and solicits value judgments related to 
managerial decisions. [SOCIAL] 

The CD-ROM supports Gagne’s instructional events 
through the use of multiple media, drill and practice. 



tutorials, and simulations. It provides a high level of 
interactivity, and immediate feedback. The use of ‘drill 
and practice’ and simulation solicits user input, 
maintains the presentation dynamic and, thus, motivates 
the users. Using the Harvey-ball format, examples of 
multimedia features that meet instructional conditions 
are presented in Figure 4. [LEARNING] 

Evaluation of Supporting Delivery Materials 
(Textbooks) 

The textbooks (Frame, 1994 and Frame, 1995) are 
among the most popular (number of sales, and 
reviewers’ comments) in the project management 
literature. They are similar in design and layout features 
and they both address complementary topics. They use 
diagrams and drawings to reinforce understanding, offer 
several examples, and occasionally include mini-case 
studies to foster reflection and application. The case 
studies present problems, and offer solutions at the same 
time, to emulate a mechanism for feedback provision. 
[STRENGTHS] 

The textbooks offer a table of contents, but lack outlines 
and glossaries. There are no special features associated 
with the textbooks (lack of mnemonics or other 
attractors). Instructor teaching materials (such as 
overhead transparencies, packaged in a notebook that 
guides comprehension and reinforce visualization and 
synthesis) are available. [ORGANIZATION] 

The books cover a broad range of topics. Reading 
should be sequential in Frame (1995), which mimics the 
phases of the project life cycle. Readers can follow a 
non-linear approach (skipping or rearranging chapter 
reading order) in Frame (1994). There is a balancing 
freedom of the instructor to intervene, as directions, 
pace, and outcomes are easily determined by the readers. 
[INSTRUCTOR] and [USERS] There are no chapter 
summaries, but each section offers conclusions and 
chapter introductions addressing main points. No study 
questions are included in the textbooks, but case studies 
and examples can be used to guide study and reflection. 
The textbooks use a variety of visuals. 
[MANAGEMENT] 

The graphics are accurate, and relevant to the objectives. 
They are likely to maintain user interest, since they 
provide examples of the diagrams, software tools, and 
cost curves discussed in the body of the textbooks. The 
graphics can be easily comprehended by reading the 
supporting explanations; they are simple and present a 
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FIGURE 4 

INSTRUCTIONAL EVENTS IN MULTIMEDIA 



Gagne’s 

Instructional Event 


Acti on in Multim e dia 


CD-ROM 


1 . Gaining attentio n 


Show the user an attractor event. Text or graphic 
animation, audio clips to engage learners 


o 


2. Informing the learner 
of the lesson objectives 
and activating 
motivation 


Questions or statements identifying leaming objectives. 
Textual reinforcement, gr^hics displaying topic 
organization, links to objectives and sub- content of each 
section 




3. Simulating recall of 
1 prior learning 


Review topics relevant to the lesson. Simulations provide 
practice and experimentation with the drills to be learned 
Text, graphics, and video soliciting recall of prior topics, or 
repeating key points 


o 


4. Presenting the stimulus 
material 


Use tutorials to introduce the content material in a variety 
of m e di a form ats. U se se quenc ed instruct! on. 




5. Providing learning 
1 guidance 


Show examples; offer overview of navigation, and 
overview of learning module. Graphics, text and video are 
suitable. 


• 


t 

6. Eliciting performance 


Use drill & practice; leading learners through practice 
exercises. U se questions and solicit actions. Appropriate to 
solicit accuracy. 


• 


i 

|7. Providing foedb ack 


Provide guidance on actions and selections. Check 
correctAncorrect answers, and offer reinforcement Any 
media format. 


• 


8. Assessing peifo nuance 

1 


Provide immediate- auto mated re suits. Encourage further 
reinforcement through text, gr^hics, and sound. Show 
mistakes and offer remedial support. 


• 


9. Enhancing retention 
and leaming transfer 


U s e hi ghly interactive acti vitie s. U se re dundant m edi a to 
repeat content in different formats. Show examples and 
stimulate concept mapping 


• 



Adapted From Gagne (1992) andShambaugh& Magjiaro (1997, p.202, 206) 



Legend: Very Low 




V ery 



unified design. The layout of the text is consistent. It is 
accurate, and generally up to date. The reading level is 
colloquial, and the topic presentation is usa*-friendly 
and easy to read. The text is likely to maintain readers’ 
interest, provided that readers have an interest in 
learning the materials. [MOTIVATION]. The study of 
the materials is designed as an independent, self-paced 
experience, and does not involve working in teams. 
[SOCIAL] 

The textbooks used in this study comply with only a few 
(3) of Gagne’s nine instructional events (Figure 5). 
They succeed in gaining attention because of the writing 
style, and the use of visuals. However, there is no 
elaboration of the learning objectives of each chapta*, 
nor any elaboration of how they are accomplished. 



Except for introductory chapter statements, there is no 
formal mechanism to summarize and stimulate recall of 
prior learning. There is no formal mechanism to assess 
performance, provide feedback or encourage users to 
complete the case studies, and build upon the 
information learned. 

Although diagrams are used to facilitate explanations of 
complex procedures, there are no particular attractcxs 
(cues and mnemonics) that stimulate retention. Readers 
need to create their own concept maps and diagrams to 
establish the links within the materials presented. The 
cognitive load is rather high, regardless of the high 
degree of coherence of the material presented. The 
textbooks do succeed in providing learning guidance, 
and presenting stimulus materials through the examples, 
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FIGURE 5 

INSTRUCTIONAL EVENTS IN TEXTBOOK INSTRUCTION 



Ciagiie's 

instructional Event 


Action In textbook instruction 


Textbooks 


1. Gaining attention 


Demonstration; Presentation style, examples 


• 


2. Informing the learner of the lesson 
objectives and activating motivation 


Visual organizers, pointers, list of 
objectives/challenges 


o 


3. Simulating recall of prior learning 


Summaries; recall questions; review of previous 
lessons 


o 


4. Presenting the stimulus material 


Use of visuals; demonstration and testimonials 




5. Providing learning guidance 


Textbooks provide examples; uses real-world 
scenarios; presents solutions 


• 


6. Eliciting performance 


Activities and exercises; workbooks; individual and 
group activities, additional assignments 


o 


7. Providing feedback 


Homework solutions; other feedback 


o 


8. Assessing performance 


Self-evaluation strategies; contrasted with solutions 


o 


9. Enhancing retention and learning 
transfer 


Repeat topics with diverse examples; encourage to 
apply skills to real situations 


0 



Adapted From Gagne (1992) and Shambaugh & Magliaro (1997, p.202, 206) 

Legend: Very Low 




to 



Very High 




cases, and anecdotes used throughout the topics 
presented. [LEARNING] 

Evaluation of Face-to-Face Instruction 

The entire project management course is designed to 
teach project management skills through case studies 
and hands-on exercises, and offers a broad variety of 
examples immediately applicable to realprojects. There 
are discussions of the entire project life cycle, from the 
definition of project requirements (early stage), to the 
development of work-breakdown structures, to project 
change and closeout. [ORGANIZATION] 

The lecture presentations are supported by the use of 
overhead transparencies. The transparencies following 
the organization of Frame (1994, 1995) display the 
graphical images of the textbook, and develop additional 
graphical representations. [STRENGTHS] 

The instructor primarily uses markers to write on 
transparencies during the presentation. Occasionally, 
display boards are used to sipport explanations. The 



instructor offers several examples, asks frequent 
questions of the audience, provides feedback to 
responses, and encourages participation through face-to- 
face discussion of section problems. [INSTRUCTOR] 
and [MANAGEMENT] 

Frequently, cases and other exercises are completed 
face-to-face, with limited instructor support. Students 
work in teams and/or independently to solve case 
studies, and to answer specific exercises. [USERS] and 
[SOCIAL] 

Face-to-face instruction uses a variety of instructional 
models. The model suitable for teaching basic skills 
(which is the objective of the introductory project 
management course-modules used in this study) is direct 
instruction. However, since the instruction is offered in 
short-modules (specific time-constraint, expectations, 
and audience), it is also comparable to workshop 
models. Therefore, the characteristics of direct 
instruction and workshops are jointly considered in the 
evaluation of the face-to-face instruction in project 
management summarized in Figure 6. 
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FIGURE 6 

INSTRUCTIONAL EVENTS IN DIRECT INSTRUCTION AND WORKSHOPS 



Gagne*s 

/nsfrucfional Event 


ActUm In Direct Instruction 


Action in ivorkshops 


FaccAo- 

face 


L 


Gaining attention 


Demonstration 


Presentation style, examples 


• 


~ 


Informing the learner oj 
the lesson objectives and 
activating motivation 


Writing on board, verbal or 
handout 


Visuals, organizers, handouts, 
materials mailed in advance 


• 


3, 


Simulating recall of 
prior learning 


Teacher or student summary; 
questions 


Addressing background of 
participants, review of previous 
lessons 


<* 


4. 


Presenting the stimulus 
material 


Teacher, student, media 


Presentation by leader or guest or 
team; demonstration, testimonials 




5. 


Providing learning 
guidance 


Teacher provides examples on 
board, student attempts 
examples, teacher walk- 
around 


Discussion, examples 


• 


e. 


Eliciting performance 


Class suggestions; individual 
prompting 


Individual activity in a workbook, 
group activities, volunteer activity; 
outside assignments 


0 


7. 


Providing feedback 


Class examples; lab activities; 
homework; verbal comments 
on work 


Providing consultation after 
workshop 


0 


& 


Assessing performance 


Comments, grading 


Criteria provided by presentation, or 
in handouts 


• 


9. 


Enhancing retention 
and learning transfer 


Repeat in subsequent lessons; 
provide additional diverse 
examples 


Material to take home 


0 



Adapted From Gagne (1992) and Shambaugh & Magliaro ( 1 997, p.202, 206) 



Legend: Very Low to Very High 



The face-to-face instruction completely meets several of 
the learning conditions, including the provision of 
assessment. Areas that need particular improvement are 
related to eliciting performance throughout the session, 
as well as including more opportunities for feedback. A 
method that may work in the direction of eliciting 
performance and enhancing retention might involve the 
presentation of materials from different sources, which 
would simultaneously improve the instructional event 
Presenting the stimulus material'' currently evaluated 
at 50% of its full potential. The instructor might ask 
students to take the lead in presenting specific case 
findings, use additional guest speakers from industry, or 
resort to video testimonials. 

SUMMARY 

The summative evaluation of the instructional materials 
shows that CBT and face-to-face instruction are two 
environments complying with a high number of 
pedagogical and technical requirements for learning 
(Figure 7). The CD-ROM offers a variety of media and 
content representations that are key ingredients of 
learning. The face-to-face instruction is supported by 
visuals, communication skills, interaction strategies and 



delivery structures that keep the learners engaged in the 
content. Textbooks, which are ranked as the least 
compliant with Gagne’s instructional events, lack the 
interactivity and feedback features that support effective 
learning and, as such, should be mainly used as 
reference to instruction, with a well defined role of 
supplementing learning. 

Overall, the comparative results indicate that CD-ROM 
and Face-to-Face instruction each satisfy more than 70% 
of the instructional event in Gagne’s model. Textbook 
instruction only meets 33% of the conditions of 
learning. The Project Management example of this 
review illustrates that independent, text-based learning 
(self-taught) may not be effective without strengthening 
mechanisms for feedback provisions, performance 
testing and reinforcement procedures (i.e., better 
clarifying lessons objectives and evaluating their 
achievement by eliciting knowledge transfer to similar 
problems/scenarios). 

DISCUSSION 

The results of this evaluation exercise specifically refer 
to the content / materials used in the introductory Project 
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FIGURE 7 

SUMMATIVE EVALUATION RESULTS 



(Jagne \ (1992) 
Instructional II vent 


CD-ROM 


Textbooks 


Face-to-face 


1. Gaining attention 


3 


• 


• 


2, Informing the learner of the 
lesson objectives and activating 
motivation 




o 


• 


3. Simulating recall of prior 
learning 


3 


o 




4, Presenting the stimulus 
material 


3 




3 


5, Providing learning guidance 


• 


• 


• 


&, Eliciting performance 


• 


o 


0 


7. Providing feedback 


• 


o 


3 


S. Assessing performance 


• 


o 


• 


9, Enhancing retention and 
learning transfer 


• 


0 


3 


Avg. % of Instructional Event Met 


86% 


33% : 


72% 



Adapted From Gagne (1992) and Shambaugh & Magliaro (1997, p.202, 206) 

,o 



Legend: Very Low 



to Very High 



Management course. However, the same approach and 
assessment model can be easily extended in other 
subject areas. Undertaking a similar evaluation exercise 
prior to implementing any program is one way to assess 
which instructional delivery elements are most/less 
effective for enhancing the learning process and 
cognitive impact. 

The actions listed in the evaluation tables for each 
instructional delivery mode (see Figures 4, 5, 6) provide 
suggestions on filling the program gaps. For example, 
to better ''elicit performance"" when using textbooks, 
faculty could require students to test their knowledge 
through: 

• Using specific exercise/reading workbooks 

• Completing case assignments 

• Accessing students’ resources web support pages 
(usually available with key textbooks) 

• Opening real/on-line discussion on specific book 
chapters; etc. 

This type of evaluation process may be useful in 
textbook selection, CRT evaluation or improving face- 
to-face instruction. Examination of textbooks is a yearly 
process in many disciplines and guidelines for 



acceptance may be created from this framework. 
Developers of CRT are creating software that recognizes 
the expertise level of the user, allowing them to by-pass 
subject matter already mastered, thereby avoiding 
tedious repetition. As mentioned earlier, faculty may 
incorporate presentations from different sources: guest 
speakers and student presentations or research. In order 
to successfully address all of Gagne’s criteria, it may be 
appropriate to include all three different instructional 
delivery modes. 

Following an evaluation approach based only on the 
delivery medium and its characteristics does not provide 
the same level of feedback that is key to foster cognitive 
engagement with the materials. A classification of 
different types of programs on intensiveness/ 
interactivity or media characteristics provides 
information only on criteria such as technical quality, 
usability, navigation, number and type of media used; 
etc. For example, using Wirth and Amos (1996) 
intensiveness scale, one would rely on aclassification of 
classroom instruction on an absolute rank of VI (see 
Figure 8), of correspondence course (a proxy for 
textbook-based) as rank IX, and of computer-based 
multimedia training (CRT) as rank X. 
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FIGURE 8 

SUMMARY EVALUATION OF PROGRAMS INTENSIVENESS ON SELECTED ENVIRONMENTS 



Program 


Instructor 
Location proximity 


Instruction 

period 


Interaction 

frequency 


Interaction 

duration 


Interaction 

type 


Total 

Score 


Delivery 

Tech- 

nology 


Rank 

order 


Classroom instruction 


3 5 


2 


4 


3 


5 


22 


FF 


VI 


Correspondence course 


1 4 


1 


2 


2 


5 


15 


MM 


IX 


Computer-based (CBT) 


1 2 


1 


2 


2 


4 


12 


CM 


X 



Legend: FF~ Face-to-Face MM= Multiple media (not necessarily synchronized) CM= Computer-mediated 
Scale: I (low)~> 3 (medium) -> 5 (high) 



Results from Figure 8 lead to different conclusions 
focused on Gagne’s framework. This is obviously based 
on the use of different parameters as proxies for 
effectiveness. However, in the learning process, 
cognitive impact of instructional programs is the key 
objective, and the use of the best delivery mode that 
supports this objective, is only one of the means to an 
end. Therefore, it is critical to focus on the elements 
that directly impact learners’ cognitive processes. 
Results from the ‘program intensiveness’ analysis give 
no or very limited information on which specific 
remedial actions should be put in place to improve 
learning. Evaluation of content organization based on 
Gagne’s criteria provides information in a manner that 
easily tracks gaps and stimulates reinforcement (i.e., 
remedial action in objectives definition), retrieval (i.e., 
simulating recall of prior learning experiences), and 
guides the learning process as well as its transfer. 
Therefore, benchmarking different programs and 
delivery environments against these criteria can be an 
important and useful tool to guide the organization of 
any lesson content, a specific module, or an entire 
course. 

ENDNOTES 

‘The intensiveness scale (considered as a proxy for 
effectiveness) is built on a set of variables: d)Instructor 
location: “High” intensiveness when instruction is 
delivered at the location of the project team, “medium” 
when it is held at conference or university premises; 
“low” when it is delivered at a distance (videotape, 
audio, correspondence), h) Instructor proximity: “High” 
intensiveness in face-to-face instruction; “low” in 
remotely guided self-teaching; and “medium” in 
multimedia and computer mediated (electronic mail) 
systems, c) Instruction period: “High” intensiveness 
when concentrated in l-5day seminars; “low” in formal 
academic degree programs. Intensiveness is inversely 



associated with the duration of instruction (with shat 
instruction segments being more intensive). d) 
Interaction frequency: Refers to the frequency of 
instructor/student sessions, with daily sessions being 
more intensive than sessions meeting only once a week. 

Interaction duration: “High” interaction isassociated 
with 5-8 hr sessions, and “low” interaction is associated 
with 1-2 hr sessions, f) Interaction type: Two-way 
interaction (student-instructor) in a classroom is the 
higher level, whereas one-way video is the lowest level. 
The two-ways automated interaction of expert systems 
and multimedia is associated with medium scores. 

^Wirth and Amos (1996) intensiveness scores use three 
measures of interactivity (frequency, duration, type), 
with the highest score given to “high” interactivity (5 
points), and lower score to “medium” (3 points), and 
“low” interactivity (1 point). 
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APPENDIX A 

NCTM SOFTWARE EVALUATION CHECKLIST 



PROGRAM NAME: Managing Projects in Organizations SOURCE: ESl-lntemational COST: $$$$ 

SUBJ ECT AREA: Project Management REVI E WE R’S N AME : xyz DATE: DD/MM/YYYY 



1. INSTRUCTIONAL RANGE 

_Business audiences Grade level(s) 

JBasic computer and basic mgt skills_ Abi I ity I eve I (s ) 

2. INSTRUCTIONAL GROUPING FOR PROGRAM USE 

X Individual 



X Small group (size_2 ) 

Large group (size ) 

3. EXECUTION TIME 

_40 X unit minutes (estimated) for average use 

4. PROGRAM USE(S) 

Demonstration 

X Drill or practice 

Instructional gaming 

Instructional management 

Instructional support 

X Problem solving 



Programming utility 

_X Simulation 

Testing operations 

Tutorial 

Whistles and bells 

X Word processing 

Other ( ) 



5. USER ORIENTATION: INSTRUCTOR’S POINT OF VIEW 
Low High 

X . . Flexibility 

X Freedom from need to intervene or assist 



6. USER ORIENTATION: STUDENTS POINT OF VIEW 
Low High 




X 

X 

X 



X Quality of direction (clarity) 

X Quality of output (content and tone) 

Quality of screen formatting 
Freedom from need for external Information 
Freedom from disruption by systems errors 
Simplicity of user input 



7. CONTENT 

Low High 

X Instructional focus 

X Instructional significance 

X Soundness or validity 
X Compatibility with other material 
used 

8. MOTIVATION AND INSTRUCTIONAL STYLE 



Passive 


Active 








X 


Type of student Involvement 


Low 


X 


High 


Degree of student control 


none poor 




good 








X 


Use of game format 






X 


Use of still graphics 






X 


Use of animation 






X 


Use of color 






X 


Use of voice input and output 


X 






Use of non-voice audio 


X 




! X 


Use of light pen 

Use of ancillary materials 






X 


Use of feedback 


9. SOCIAL CHARACTERISTICS 




Present and 


Not 


Present and 


negative 


present 


positive 




X 




Competition 




X 




Cooperation 




X 




Humanizing of computer 






X 


Moral issues or value judgment 






X 


Summary of students 


performance 
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CONCEPT MAPS AS AN ALTERNATIVE 
TECHNIQUE FOR ASSESSING STUDENTS’ 
UNDERSTANDING OF TELECOMMUNICATIONS 
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ABSTRACT 

An alternative method of measuring the knowledge of students is to use mental models, and specifically concept maps. 
Concept maps provide a visual representation of conceptual and relationship knowledge within a particular domain. 
This study builds on previous studies and attempts to reduce their methodological weaknesses. Students in several 
undergraduate sections of an MIS Telecommunications course were asked to create concept maps of their 
Telecommunications knowledge at three distinct points throughout thesemester. These concept maps were compared 
across all students at a single point in time, within students across the three time periods, and against the concept 
map of a domain ^'expert” at both the individual and composite levels. Findings indicate that the individual and 
composite maps increased significantly in size over time, and comparisons with the ''expert” map show a 
significantly increasing overlap of concepts overtime. Moreover, this study shows the applicability of this technique 
as an alternate assessment method. 



INTRODUCTION 

Information systems (IS) educators are tasked with 
preparing students with a broad education in business 
and IS, but measuring students’ total knowledge can be 
difficult. Throughout a semester, students complete 
quizzes, tests, homework, and other assignments to 
demonstrate their mastery of the particular topic or 
domain of concern. When taken together, these 
measures of knowledge tell how much students know, 
and just as important, what they do not know. Still, 
most of these measures are very structured and limiting 
to the students— they may know 85% percent of the 
course’s material, but the test, quiz, or homework 
assignment may only cover 50% of the material, 
including the 15% they did not know (see Figure 1). 
Moreover, these traditional techniques do not allow the 
students to demonstrate knowledge and mastery beyond 
the assessment technique. To increase knowledge 
measurement accuracy, these traditional assessment 



formats should be supplemented with alternative 
approaches. 

FIGURE 1 

TRADITIONAL ASSESSMENT 
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One alternative approach to knowledge assessment is 
the use of mental models, and specifically concept maps 
or concept webs. When a student creates a visual 
representation of his/her cognitive conceptualization of 
the IS field, viewers of that map get an inside look into 
that student’s mind. The concepts and their 
relationships to each other are represented visually, 
showing the items that the student knows, their 
relationships, and the items that the student does not feel 
are important enough to be included. 

The purpose of this research is to illustrate a potential 
use of concept maps as an assessment tool of students’ 
conceptual knowledge of the IS domain (though 
applicable to other domains), and specifically 
Telecommunications. For the purposes of this research, 
Telecommunications (Telecomm) includes topics such 
as the Internet, networking, cabling, and 
communications, among others, and the regulations that 
govern them. 

RESEARCH QUESTIONS 

Previous studies involving concept maps and the field of 
information systems were limited in their findings (e.g.. 
Freeman and Urbaczewski, 2001). This was mostly due 
to the methodology and a focus on concept maps drawn 
over a five-week period, and not as a snapshot within a 
short period of time. These studies did not utilize 
concept maps in a traditional assessment scenario, and, 
therefore, a clear insight into the usefulness and merits 
of concept maps has not been provided and there are a 
number of questions left to be answered regarding the 
use of concept maps as assessment techniques. This 
study attempts to answer the following research 
questions. 

1. How does the knowledge of individual students 
change over time? 

2. How does the composite knowledge of students 
change over time? 

3. How does the knowledge of individual students 
compare to that of the “expert” over time? 

4. How does the composite knowledge of students 
compare to that of the “expert” over time? 

The analyses will provide insight into several issues. 
First, the concept maps will show what the students 



know (or at least feel to be important) regarding 
Telecommunications. Second, comparisons of the 
students’ maps to an expert’s map will provide 
information regarding how much is learned from the 
course and whether the concepts that are learned and 
included in the maps are done so “correctly” and as 
intended according to the expert ~ the faculty instructor. 
Third, comparisons of the three concept maps of 
individual students will provide a measure of how the 
student’s knowledge of Telecommunications changes 
over time. Finally, if there are major differences 
between the student maps and the expert map, changes 
may be necessitated in the teaching of the 
Telecommunications class(es) so that the students have 
closer conceptualizations to the expert. These concept 
maps could potentially be used as an overall assessment 
of the department’s teaching efficacy or simply as a 
view into the students’ minds. 

PRIOR LITERATURE 

The use of mental models, specifically concept maps, 
can aide in the assessment of knowledge at a conceptual 
level (Fisher, 1990; Fisher et al., 1990; Gaines and 
Shaw, 1995; O’Neil and Klein, 1997). Concept maps 
provide a visual representation of conceptual and 
relationship knowledge of main concepts and major sub- 
topics within a particular domain (Hoover and Rabideau, 
1 995). Concept maps consist of nodes that represent the 
concepts and optionally labeled arcs that connect the 
nodes and represent the presence of a relationship. 
Concept maps look like a spider’s web consisting of 
many concepts or nodes connected to each other by lines 
signifying the presence of relationships. Concept maps 
are typically assessed by comparing them to an expert’s 
map in either quantitative or qualitative forms. Other 
assessment techniques iix:lude counting the number of 
concepts, the number of relationships, and the number 
of interconnections to determine the degree of 
complexity. See Figure 2 for a sample concept map. 

Other forms of mental models such as cognitive maps, 
semantic networks, and schemata can also be used to 
represent relationships between concq)ts. However, 
they differ from concept maps in that they also include 
directionality to the relationship or causality between the 
concepts (Fisher, 1990). Concept maps were chosen as 
the assessment method because they do not require any 
temporal or cause/effect relationships between the 
concepts. In addition, concept maps support the notion 
of hierarchy, though they do not force a hierardiy. 
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FIGURE 2 

SAMPLE CONCEPT MAP (TABER, 1994) 




A general concept map showing the mapping of 
“concept maps” according to Taber. Note that the links 
are displayed as arrows showing the direction of the link 
(not causation), a convention not always used by others. 

Concept maps have been used for many years as a 
means for communicating knowledge in fields such as 
education, biology, history, mathematics, engineering, 
computer science, and communications (Clibum, 1986; 
Gaines and Shaw, 1995; Wallace andMintzes, 1990; 
Williams, 1995). As seen in Figure 2, concept maps are 
an alternative to linear text, they help reveal 
misconceptions, and they increase learning. Freeman 
and Urbaczewski (1999) describe the use of concept 
maps as an “end-of-[academic] major” assessment 
technique to measure students’ IS knowledge gained 
throughout their coursework. While there may be merit 
in creating and implementing some sort of assessment 
measure to be given to students after they complete all 
of their coursework, this would involve major cultural 
and pedagogical changes and is unlikely to happen at 



most institutions in the near term. Even so, this study 
was an initial attempt at using concept maps as an IS 
educational assessment technique. However, as 
described in Freeman and Urbaczewski (2001), there 
were a number of limitations and weaknesses associated 
with that first study. 

First, there were no comparisons within individuals over 
time. Research suggests that concept maps indeed show 
the differences between novices and experts within a 
field (Markham et al., 1994; Wallace and Mintzes, 
1990), so future studies should analyze the differences 
in concept maps from individuals drawn at different 
points in time. A second weakness was that the 
assignment covered a five-week period, as opposed to 
one or more snapshots in time. Concepts maps drawn 
over five weeks will likely be different from concept 
maps drawn within a time period of several hours, or 
shorter. Third, while the study focused on individual 
students and their individual concept maps, there was no 
control over the students to assure the researchers that 
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there was no outside collaboration by the students 
during the five-week time period. 

Finally, it should be noted that students have typically 
found concept maps to be fun (Freeman and 
Urbaczewski, 1999; Taber, 1994), an emotion unlikely 
to be expressed with regard to many other traditional 
assessment methods like exams and quizzes. 

RESEARCH METHODOLOGY 

This current study takes the composite knowledge of the 
above literature review and, with a focused effort based 
on the concept mapping literature from other disciplines, 
attempts to eliminate the weaknesses of this earlier work 
within IS. 

Eighty-five students in an undergraduate MIS 
telecommunications course at a large U.S. university 
were the subjects for this study. At three different times 
throughout the Spring 2001 semester — during the first 
week of class, during the eighth week of class, and 
during the last week of class — a third party (not the 
instructor) visited the classes and witnessed the students 
as they created concept maps centered around 
Telecommunications. Priorto the first visit, the students 
were given a brief ( 1 5 minutes) introduction and training 
session on the subject of concept maps, how they can be 
drawn, and what they can show the reader. The students 
were given a blank sheet of paper to use in creating their 
concept maps, and they were given 20 minutes to create 
their maps. Students were not permitted to look at 
earlier maps in creating a current map, nor were they 
allowed to keep copies of the maps that they created 

As an incentive to participate, any student that 
completed all three concept maps during the semester 
received three bonus percentage points towards his/her 
overall course grade. Sixty-two students completed all 
three maps, and those sets of maps are the objects of 
analysis. 

ANALYSIS 

Analysis of the concept maps consisted of the following: 

Coding Scheme 

Each map was redrawn verbatim, but only focusing on 
and including the concepts and relationships regarding 
Telecomm. The cutoff for inclusion was to not include 
any concept and the associated relationships if that 



concept would not be included in a typical textbook 
chapter on telecommunications or internetworking. 
Therefore, “web-enabled database” would be included, 
but its relationship to “SQL” and “SQL” as a concept 
would be beyond the scope of Telecomm. Two outside 
coders, unaffiliated with the study, completed these 
redraws and the rest of the coding process to prevent any 
undue biases on the part of the authors from creeping 
into the analysis. Both authors independentlycompared 
the redraws for consistency and for concept 
categorization. 

Following these redraws, the maps were randomly split 
into two groups. Using the maps from Group A, a set of 
consistent terms was developed to create a consistent 
coding scheme for all of the maps. In other words, if 
one map contained “employee” and another map 
contained “worker,” or if one map contained “cabling” 
and another map contained “wiring,” the coding scheme 
was necessary to select one term to be used in all 
instances of these synonyms to keep all of the maps 
consistent with each other. This coding scheme was 
then applied to Group B, and the maps from Group B 
were redrawn using the new coding scheme. Any 
concepts that were part of Group B, but not included in 
the coding scheme, were added to create a more robust 
scheme. The maps from Group A were then redrawn 
using this revised coding scheme. This process was 
repeated until all concepts from Groups A and B were 
accounted for in the coding scheme and all of the maps 
had been redrawn using the finalized coding scheme. 
Once the maps were redrawn with the finalized coding 
scheme, each concept and relationship from every map 
was entered into a software package for manipulation 
and analysis. 

General Statistics 

The final coding scheme contained 75 1 unique concepts, 
though there were over 1,600 total concepts (when 
including synonyms) used by the students. The 186 
maps (62 students x 3 maps each) contained 5,000 total 
concepts in 2,606 unique relationship pairs, and over 
5,100 total relationships. Across all maps from all 
rounds, there was a mean of 27 concepts and 28 
relationships on each map. The smallest map had 7 
concepts and the largest had 50. These same two maps 
also anchored the range of relationships with 6 
relationships and 55 relationships, respectively. Of all 
186 maps, only three had less than 1 0 concepts, and only 
6 had less than 10 relationships. 
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Individual Maps 

Prior research as well as everyday classroom experience 
indicates that the student concept maps should increase 
in size over time, representing greater domain 
knowledge. Fifty-seven (out of 62) students had larger 
maps in the second round than in the first round. 
Additionally, 49 students had larger maps in the third 
round than in the second round For 48 students, their 
third round map was the largest of the three rounds, and 
for 44 students, each round’s map was larger than the 
previous round. Overall, therefore, the maps of 
individual students did increase in size throughout the 
study. 

Composite Maps 

More details regarding the changes in the individual 
maps over time can be learned from analyses of the 
composite maps from each round. Table 1 presents 
descriptive statistics for each round. 



TABLE 1 

DESCRIPTIVE STATISTICS 
FOR COMPOSITE MAPS 





Round 

1 


Round 

2 


Round 

3 


P- 

value 


Unique Concepts 


380 


418 


459 


n/a 


Total Concepts 


1175 


1742 


2083 


n/a 


Unique 

Relationships 


841 


1133 


1271 


n/a 


Total 

Relationships 


1196 


1789 


2147 


n/a 


Mean Concepts 


18.95 


28.10 


33.60 


<0.001 


Mean 

Relationships 


19.29 


28.89 


34.63 


<0.001 


Complexity 


1.34 


1.79 


2.03 


0.127 



Overall, the maps were significantly larger in each 
subsequent round in termsof concepts and relationships. 
They also increased in complexity in each subsequent 
round (a measure of the number of relationships 
depicted in the map beyond the minimum necessary to 
connect all of the concepts linearly), though not at a 
significant level. 



Based on the mean number of concepts and the mean 
number of relationships in each of the three rounds and 
overall, the following four tables (Table 2-Table 5) 
present the most common concepts and relationships for 
the respective rounds and overall. More concepts and 
relationships are included in all four tables in order to 
accurately represent the frequency distribution and not 
omit concepts or relationships with the same frequency. 
In other words, while Round 1 had a mean of 19 
concepts, the top 22 concepts are shown in Table 2 to 
complete the frequency of all concepts appearing on 12 
maps. Similar adjustments have been made to the 
concepts and relationships in Tables 2-5. These 
“amended means” will also be used below for 
comparisons to the expert’s map. 

The Expert’s Map 

In addition to the student maps, an independently-drawn 
concept map of an “expert” — the faculty member who 
taught these sections of Telecommunications — was 
collected for analysis and comparisons. The expert’s 
map contained 86 concepts and 103 relationships, giving 
a complexity score of 18. For comparison, the largest 
student map contained 50 concepts and 55 relationships 
(also the maximum), and this map was in the third 
round. The maximum complexity score for any of the 
student maps was 10, but this was in the first round. 
Therefore, without any additional comparisons and 
analyses, it is already quite apparent that the expert’s 
map is much larger and more complex than any map 
drawn by the students. 

Even so, comparisons can be made with the individual 
maps against the expert’s map, and with the composite 
maps and summary round data against the expert’s map. 
Every student map had at least one concept in common 
with the expert’s map, and some maps had up to 23 
concepts in common. While these 23 concepts only 
represent 27% of the concepts on the expert’s map, they 
represent 64% of the student’s map. This indicates that 
while nearly every msp was much smaller in size than 
the expert’s map, some of the maps contained a 
substantial amount of overlap in terms of the concepts 
included. 

Table 6 shows the number of concepts, on a composite 
level, contained in both the students’ maps and the 
expert’s map. The increasing overlap with the expert’s 
map in each subsequent round was significant for all 
three measurements — actual number of overlapping 
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TABLE 2 

ROUND 1 MOST COMMON CONCEPTS AND RELATIONSHIPS 



Concept 


# of Maps 


Relationship 


# of Maps 


Telecommunications 


63 


Telephone:Telecommunications 


23 


Telephone 


34 


TelecommunicationsComputers 


18 


Internet 


33 


Telecommunications:Networks 


18 


Computers 


32 


Telecommunications:Intemet 


17 


Cellular 


30 


TelecommunicationsiBusiness 


12 


Networks 


24 


Telephone:CeIlular 


11 


Email 


22 


TelecommunicationsCommunication 


10 


Business 


21 


Telecommunications:Fax 


9 


Cabling 


21 


TelecommunicationsTechnology 


9 


Individuals 


16 


Telecommunicationsilndividuals 


9 


LAN 


16 


Wireless:Telecommunications 


9 


Satellites 


15 


Telecommunications:Cellular 


7 


Wireless 


15 


Telecommunicationsilnformation 


7 


Fax 


13 


Networks:LAN 


7 


Networking 


13 


WAN’.Networks 


7 


Video Conferencing 


13 


Telecommunications: Satellites 


7 


WAN 


13 


Video Conferencing:Telecommunications 


6 


AT&T 


12 


Television:Telecommunications 


6 


Modem 


12 


TelecommunicationsCabling 


6 


Pagers 


12 


IntemetiComputers 


6 


Speed 


12 


Telecommunications: Speed 


6 


Technology 


12 


Networks:Computers 


6 






Telecommuting:Teleoommunications 


6 






TABLE 3 






ROUND 2 MOST COMMON CONCEPTS AND RELATIONSHIPS 




Concept 


# of Maps 


Relationship 


# of Maps 


Telecommunications 


63 


Telephone.Telecommunications 


22 


Telephone 


40 


Wireless:Telecommunications 


19 


Wireless 


35 


TelecommunicationsNetworks 


16 


Fiber Optic 


32 


Transmission Media.Fiber Optic 


13 


LAN 


32 


Transmission Media:Tdecommunications 


13 


Protocols 


27 


TelecommunicationsComputers 


12 


WAN 


27 


TelecommunicationsNetworking 


11 


Internet 


26 


Telecommunications:Intemet 


11 


TCP/IP 


26 


TelecommunicationsProtocols 


11 


Transmission Media 


26 


TCP/IP:Protocols 


11 


Cellular 


25 


Networks :LAN 


11 


Star 


25 


Wireless:Cellular 


11 


Network Topology 


24 


Twisted Pair:Transmission Media 


10 


Networks 


24 


Telecommunications:LAN 


10 


Cabling 


23 


WAN:Networks 


10 


Twisted Pair 


23 


Star:Network Topology 


10 


Analog 


20 


TelecommunicationsNetwork Topology 


9 


Data 


20 


Network Topology:Bus 


9 


Email 


20 


Telephone:LATA 


8 


Modem 


20 


Telecommuting:Teleoommunications 


8 


Bus 


19 


TelecommunicationsData 


8 


Digital 


19 


Networking:LAN 


7 


Computers 


18 


TelecommunicationsData Sharing 


7 


Ring 


18 


Ring:Network Topology 


7 


LATA 


16 


Wireless:PDAs 


7 
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Table 3 
(continued) 



Concept 


# of Maps 


Relationship 


# of Maps 


Microwave 


16 


WAN :Networking 


6 


Phone Companies 


16 


TCPAP:Intemet 


6 


Coaxial Cable 


15 


TelecommunicationsConnections 


6 


Ethernet 


15 


WAN:Telecommunications 


6 


OSI Model Structure 


15 


Telecommunications:Technology 


6 


Voice 


15 


WWW:Intemet 


6 






Transmission MediaiCabling 


6 






TelecommunicationsiSecurity 


6 






TelecommunicationsCommunication 


6 






TelecommunicationsiSwitched Circuit Service 


6 






TelecommunicationsOSI Model Structure 


6 






TelecommunicationsError Checking 


6 



TABLE 4 

ROUND 3 MOST COMMON CONCEPTS AND RELATIONSHIPS 


Concept 


# of Maps 


Relationship 


# of Maps 


Telecommunications 


63 


WirelessiTelecommunications 


25 


Wireless 


51 


Telecommunications:Intemet 


23 


Telephone 


44 


WirelessiCellular 


22 


Internet 


43 


Telephone:Telecommunications 


19 


Cellular 


36 


Transmission Media:Fiber Optic 


17 


LAN 


36 


TelecommunicationsNetworks 


17 


Fiber Optic 


35 


Telecommunications.Security 


17 


DSL 


32 


Security:Firewall 


14 


WAN 


32 


Twisted Pair:Transmission Media 


14 


Security 


31 


Transmission Media:Tdecommunications 


13 


Bus 


29 


TCP/IP:Protocols 


13 


Twisted Pair 


29 


TelecommunicationsiLAN 


12 


Star 


28 


StarNetwork Topology 


12 


Modem 


27 


Network TopologyiBus 


12 


Transmission Media 


27 


WAN :T elecommunications 


11 


Cabling 


26 


WirelessiPDAs 


10 


Network Topology 


25 


Telecommunicationsilardware 


10 


Networks 


25 


Networks :Network Topology 


10 


Email 


24 


Network TopologyiMesh 


10 


Ethernet 


23 


Networks.LAN 


10 


Ring 


22 


Ring:Network Topology 


9 


TCPAP 


21 


Wireless: Satellites 


9 


Digital 


20 


TelephoneiSwitched Circuit Service 


9 


Mesh 


20 


WAN.Networks 


9 


Protocols 


20 


TelecommunicationsiSwitched Circuit Service 


8 


Satellites 


20 


NICsiHardware 


8 


Video Conferencing 


20 


WirelessiBluetooth 


8 


Analog 


19 


Video ConferencingiTelecommunications 


8 


Firewall 


19 


TelecommunicationsNetwork Topology 


7 


Hardware 


19 


TelecommunicationsData Sharing 


7 


Hubs 


19 


TelecommunicationsiProtocols 


7 


Router 


17 


TelecommunicationsDSL 


7 


Microwave 


16 


Protocols:IPX/SPX 


7 


Coaxial Cable 


15 


TelecommunicationsiCellular 


7 


MAN 


15 


WirelessiMicrowave 


7 


Switched Circuit Service 


15 
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TABLE 5 

OVERALL MOST COMMON CONCEPTS AND RELATIONSHIPS 



Concept 


# of Maps 


Relationship 


# of Maps 


Telecommunications 


189 


Telephone:Telecommunications 


64 


Telephone 


118 


Wireless:Telecommunications 


53 


Internet 


102 


Telecommunications:Intemet 


51 


Wireless 


101 


TelecommunicationsiNetworks 


51 


Cellular 


91 


Wireless:Cellular 


38 


LAN 


84 


TelecommunicationsComputers 


36 


Fiber Optic 


76 


Transmission Media:Fiber Optic 


31 


Networks 


73 


Networks: LAN 


28 


WAN 


72 


WANiNetworks 


26 


Cabling 


70 


TCP/IP:Protocols 


26 


Email 


66 


Transmission Media:Tdecommunications 


26 


Computers 


63 


TelecommunicationsiLAN 


25 


Modem 


59 


TelecommunicationsSecurity 


24 


Star 


56 


Twisted Pair:Transmission Media 


24 


Transmission Media 


56 


Star.Network Topology 


22 


Twisted Pair 


55 


TelecommunicationsCommunication 


21 


TCP/IP 


53 


Telephone:Cellular 


21 


DSL 


51 


Telecommunications:Business 


21 


Protocols 


51 


Network Topology:Bus 


21 


Bus 


50 


Telecommunications:Technology 


20 


Network Topology 


50 


TelecommunicationsiNetworking 


19 


Security 


47 


Telecommuting:Teleoommunications 


19 


Digital 


46 


Security:Firewall 


19 


Satellites 


46 


TelecommunicationsProtocols 


19 


Video Conferencing 


46 


Video ConferencingiTelecommunications 


18 


Business 


43 


TelecommunicationsiCellular 


18 


Analog 


42 


WAN:Telecommunications 


18 


Ethernet 


42 


WirelessiPDAs 


18 


Ring 


42 







TABLE 6 

OVERLAP BETWEEN 
INDIVIDUAL AND EXPERT MAPS 





Round 

1 


Round 2 


Round 3 


p’Value 


Mean Overlap 
Concepts 


4.68 


10.76 


12.87 


<0.001 


Percent of 

Students’ 

Maps 


24.39% 


37.76% 


38.25% 


<0.001 


Percent of 
Expert’s Map 


5.44% 


12.51% 


14.97% 


<0.001 



concepts, percentage of the students’ maps that overlap 
with the expert’s map, and percentage of the expert’s 
map contained in the students’ maps. 

There are two additional ways to compare the expert’s 
map with the composite maps from each of the three 



rounds. The first is by comparing the most common 
concepts from each round to the concepts used by the 
expert. In Round 1 , out of the 22 (“amended mean”) 
most common concepts, only 6 (27.27%) were also a 
part of the expert’s map. For Round 2, this number 
jumps to 21 out of 3 1 (67.74%). However, the number 
falls to 22 out of 36 (61.11%) for Round 3. Still, this 
does indicate a fair amount of overlap between the 
composite maps from each round and the expert’s map. 
The second comparison is determining the number of 
concepts used by the expert that appear in any of the 
students’ maps in a given round. Of the 86 concepts in 
the expert’s map, 38 did not appear on any students’ 
map in Round 1 , 22 did not appear on any students’ map 
in Round 2, 14 did not appear on any students’ map in 
Round 3, and 6 never appeared on any students’ map in 
any of the rounds, again indicating an overall overlap 
between the students’ maps and the expert’s map, 
especially in the later rounds. 
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Course Grades 

The final analysis is the correlation between the 
students’ final course grades (based on traditional 
assessments) and their concept maps from Round 3. 
The course grades were converted to a GPA based on a 
4.0 being an A. Table 7 shows these correlations. 



TABLE 7 

CORRELATIONS OF CONCEPT MAPS 
WITH COURSE GPA 





GPA 


p-value 


Concepts 


0.376 


0.003 


Relationships 


0.370 


0.003 


Complexity 


0.044 


0.732 



The positive and significant correlations between GPA 
and both Concepts and Relationships indicate that the 
students who performed well in the traditional 
assessments also produced concept maps that were 
larger with more relationships. While the correlation 
with GPA and Complexity was slightly positive, it was 
not significant as expected based on the relatively low 
complexity of the concept maps. 

DISCUSSION 

The results are fairly clear: as the students progressed 
throughout the semester, Iheir concept maps 
significantly increased both in size and in similarity to 
the expert’s map. We would expect any other 
assessment method to yield similar results, hopefully as 
a measure of learning. 

One might wonder if the increasing number of concepts 
and relationships would be a self-fulfilling prophecy. 
That is to say, that certainly by the end of the semester, 
the students would have been exposed to more material 
and would have been able to express more of it on their 
maps. This is probably true to some extent. Of 
importance to us, additionally, is theirability to continue 
to recall basic topics that were primarily discussed in the 
beginning of the semester, as well as relate those topics 
to their understanding of Telecommunications before 
beginning the course. This demonstrates additive 
knowledge on the reshaping of prior knowledge. 

If we look at the relationship between the average 
overlap of concepts, even at its highest (12.87 concepts: 
38.25% of the students’ maps, 14.97% of the expert’s 
map), we may conclude that the transfer of knowledge 



is limited and that both the professor and students 
should be given failing grades. Is this so? We do not 
believe it is. We believe that this is rather a celebration 
of the educational system and the abilities of individuals 
to focus in areas that may not be of primary interest to 
the expert. This is the ability to inspire. Of course there 
must be some base level of knowledge in order for one 
to inspire, and we at least find that in the increasing 
number of concepts represented in the students’ maps. 
But to suggest that the goal is to have the studoits 
repeating the course material as a mantra upon 
completion would imply that our universities are no 
more than memorization camps. These maps are one 
more tool in getting inside the minds of our students and 
observing the relative importance of certain concepts. 

In the results, we find that course grade is positively and 
significantly correlated with both number of concepts 
and number of relationships in the individual maps. 
This can be attributable to a number of factors. First of 
all, and what we as educators are more likely to espouse, 
it is a demonstration of understanding of the subject 
matter. While it is true that the individual who does not 
know anything about the subject could not draw a 
concept map similar to an expert’s map, the opposite 
relationship is not necessarily true. We also cannot say 
for sure that the primary cause of the correlation is 
increased knowledge. Other mitigating factors, such as 
motivation, attitude towards course, attitude towards 
instructor, or attitude towards the researcher could all 
play a role in identifying the reasoning behind the 
correlation. For the purposes of this paper, we make no 
claim beyond the existence of positive and significant 
correlations. Future research should attempt to measure 
these differences. 

We find no relationship between grade in course and 
complexity. This is likely related to the low overall 
complexity of all studentmaps. Future research should 
also investigate the reasoning behind relatively low 
complexity and the mental model of the student. For 
example, does the student view knowledge as largely 
hierarchical, or as an interconnected web of concepts, 
and how does the student demonstrate that knowledge? 

These concept maps allow the instructor to “check” on 
what students understand and know regarding the course 
and his/her expectations. Of course, quizzes and exams 
attempt to do just that, but they are limited in what can 
be conveyed and in what can be covered. Additionally, 
beyond the quantitative analyses that appear in this 
paper, there are several valuable qualitative analyses 
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that can be performed on the concept maps. By looking 
for ‘‘strange” relationships within the concept maps, the 
instructor can clearly see what the students don’t 
understand, or at least misunderstood at one point. For 
example, if the relationships “Network Topology:LAN” 
and “Network TopologyiClient Server” appeared, it 
would be apparent that there is a misunderstanding of 
Network Topology within the class. These “strange” 
relationships are particularly helpful whenuncovered in 
the middle of the course. In the beginning of the course, 
many students will likely have misunderstandings about 
concepts and relationships merely due to the fact that 
they have not yet been covered adequately, if at all . At 
the end of the course, the instructor may not have the 
time to properly correct the misunderstandings before 
the students take part in traditional, graded 
examinations. However, inthe middle of the course, the 
instructor has not only covered a fair amount of the 
material, but also has the necessary time to clarify 
misunderstandings. 

A second qualitative analysis involves the actual 
knowledge and understanding conveyed in the concept 
maps. Ideally, as the course progresses, the concept 
maps will contain concepts and relationships that 
indicate a deeper understanding of the material. These 
concepts and relationships will go beyond the basic 
terminology of the course, and the relationships will 
become more complex. These qualitative analyses go 
beyond traditional assessment techniques in providing 
the instructor with a much clearer view of what his/her 
students know, think, and understand. 

A potential limitation with interpreting the data is that 
the researchers and coders were required to take the 
topics as they were printed on the paper. It is difficult 
from the writing to ask “is that what he/she really 
meant?” From the context and the relationships, it is 
easy to tell that in one sense “cable” may refer to a type 
of Internet access through a cable TV company, while in 
another sense “cable” may refer to a length of copper 
surrounded by insulation. However, this interpretation 
is certainly not perfect. The alternative, though, would 
be to have students pick words and definitions from an 
approved list. This would certainly defeat the purpose 
of this free association task and almost “force” the 
results. 



CONCLUSION 

This study is an attempt to better assess students’ 
knowledge using concept maps, and an attempt to 
modify previous studies and eliminate their weaknesses. 
Several issues should be noted: the contributions of this 
study and the need for additional research. First, this 
study makes several contributions to research and to 
teaching. It informs the field of IS academics in 
general, and specifically Telecomm academics, of an 
additional method for the assessment of students that 
has gained support in other academic fields such as 
education and psychology. Granted, this method may 
not be appropriate to replace other assessment methods, 
but that will depend on the specific situation and the 
type of knowledge to be assessed. This empirical study 
also demonstrates the applicability of this assessment 
method in a real classroom situation and shows what can 
be learned. 

While the concept maps of these students were 
compared to an expert, this expert was a single faculty 
member and not all of the faculty members that teach or 
have taught the Telecommunications class at this 
particular institution. Student maps could be compared 
to a “departmental” or “institutional” map or even to the 
map at a discipline level from the IS 1997 or IS 2002 
requirements (available through the AIS website, http:// 
aisnet.org/Curriculum^. Future research should also 
analyze concept maps created by groups as an 
alternative method for conveying knowledge. Another 
area for future research is the use of concept maps in 
other ways within the classroom, such as for organizing 
lectures and units, for brainstorming, or for bringing 
together diverse viewpoints. 

The optimal means for assessing student knowledge 
may often seem like the quest for the Holy Grail. While 
concept mapping may not be the “silver bullet” that is 
all-encompassing and useful in every situation, it is one 
more “fly” in the teacher’s “tackle box,” and one that 
should not be ignored. This study attempts to show the 
usefulness of this tool in one situation, and its 
applicability to many other situations can be surmised 
from this study. 
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Extended Abstract 



Nearly all MBA programs require that their graduates 
possess effective decision making skills, or a variation 
thereof, as one of the exit competencies. Y et, one would 
be hard pressed to find a required course on modem 
decision analysis in a traditional MBA curriculum. Now 
with the widespread use of spreadsheets in business and 
the newly developed decision analysis software add-ins, 
the time has come to introduce MBA students to the 
tools and techniques of this often overlooked science. 

Almost all course descriptions, regardless the subject 
matter, include the term “decision making” or “decision 
makers.” Moreover, almost all courses include 
improving managerial decision making in their subject 
area, whether it be finance, accounting, economics, 
marketing, human resources management, information 
systems management, operations management, statistics, 
or quantitative analysis, etc., as one of their objectives. 
Yet, none of these courses cover the topics of 
structuring, modeling, analyzing and solving decision 
problems. The exception is the quantitative analysis 
course, where a limited treatment of decision analysis 
using decision trees is presented. This treatment, 
however, lacks the depth needed to understand how 
decision analysis can help decision makers think hard 
about problems. It also lacks the breadth to cover the 
latest developments in decision analysis such as 
influence diagrams, value-focused thinking, and the use 
of spreadsheets for modeling and analysis. 

If MBA programs expect their graduates to be effective 
decision makers, they need to provide them proper 
training in modem decision analysis. We propose such 
training using a book entitled “Making Hard Decisions 
with DecisionTools®”, by Clemen and Reilly. 
DecisionTools is a suite of five software programs: 



PrecisionTree, TopRank, @Risk, BestFit, and 
RISKView, each designed as an add-in to Microsoft 
Excel. These programs allow the user to structure, 
model, analyze and solve complex decision problems 
using decision trees and influence diagrams. 

Elements of decision problems include 1) values and 
objectives, 2) decision alternatives, 3) uncertain events, 
and 4) consequences. According to Clemen, “decision 
analysis provides structure and guidance for thinking 
systematically about hard decisions... and provides 
analytical tools that can make the required hard thinking 
easier.” Why hard decisions? The answer is rather 
obvious. An easy decision requires little thinking and 
analysis. An easy decision is exactly that, easy! So 
what makes a decision hard? A decision may be hard 
because of its complexity, its multiple sources of 
uncertainty, its multiple and conflicting objectives, or 
because different perspectives lead to different 
conclusions. 

We can find many hard decision problems in business 
and government. Using the tools and techniques of 
modem decision analysis, decision makers can structure, 
model, analyze and solve hard decision problems, while 
taking into consideration the decision maker’s values 
and objectives, perceived uncertainties, and preferences. 

Managers and people in decision-making positions need 
to study decision analysis to make better decisions. 
Making better decisions implies that we know what 
makes a decision good. A decision may be considered 
good if it results in the best outcome. We need to 
distinguish, however, between a good decision and a 
lucky outcome. A good decision may not always result 
in the best outcome, and the best outcome may not 
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always be the result of a good decision. Using the tools 
and techniques of modem decision analysis, managers 
can be confident the decisions they make are good, 
regardless of outcome. At the very least, managers can 
learn more about the structure of their problem, and gain 
deeper understanding of their values and objectives, the 
inherent uncertainties, and their preferences. 



In the remainder of this paper, we describe the decision 
analysis process, we provide detailed descriptions of all 
tools and techniques including DecisionTools programs’ 
capabilities, and we detail some of the application areas 
relevant to business decision making. 
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Extended Abstract 



This paper reports current status of a continuing study of 
team process as applied to an undergraduate IS 
curriculum. This continuing investigation identifies and 
assesses methods for improving the team experience and 
team product in classes where teamwork is critical [1]. 
Current work focuses on two new dimensions: software 
engineering approaches and peer instruction 
interventions. 

Why do we require team projects is so many IS courses? 
Most IS educators understand that industry projects are 
seldom conducted by individuals. Well-established 
curricular guidelines require substantial experience with 
teams in IS and CS classes. Varieties in student learning 
styles mean that some students will learn the concepts 
more effectively in teams [2]. And recent work 
concerning software development indicates that pair 
programming yields high quality software in shorter 
periods of time [3]. 

Given these powerful reasons, it is no surprise that IS 
educators seek to meet the following objectives with the 
assignment of a team project [1]: 

• Increase student retention of course content such as 
methodologies. 

• Teach students howto function effectively on a team. 



• Improve the quality of project deliverables such as 
software and documentation. 

• Prepare students well for the world of professional 
work. 

This paper extends prior work by examining the use of 
software engineering techniques included in Software 
Engineering Institute’s (SEI) Team Software Process 
[4]. SEI work over the last two decades has led to the 
development of a trilogy of models including the 
Personal Software Process (PSP), the Team Software 
Process (TSP), and the Capability Maturity Model 
(CMM). TSPi represents a slightly scaled-back version 
of TSP intended for college audiences. The focus is on 
quantifiable process, highly specified roles for each 
team member, and an eight-stage iterative system 
development life cycle (SDLC). Each role is specified 
in some detail and includes a script of tasks and 
responsibilities for each of the eight stages. 

The TSPi approach to teamwork was applied in two 
upper division courses required of IS majors. The first 
course is a traditional system analysis and design 
(SA&D) course that is overflowing with course content 
such as the general SDLC, structured techniques, 00 
techniques, plus a team project intended to weave the 
concepts together. The abundance of material to cover 
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in SA&D necessitated provision of an abbreviated 
introduction to the TSP involving primarily the 
specification of each team member’s role and 
responsibilities. 

TSPi was introduced and used on a more extensive basis 
in the second course that has the traditional SA&D 
course as its prerequisite. This second course is a 
senior-level capstone course where IS majors are 
introduced to their secoixl programming language in 
order to develop a fully functional system involving 
programming, embedded SQL and graphical user 
interfaces. The first half of the term focuses on the new 
language (Visual Basic) and the second half focuses on 
a team project generating a fully functional piece of 
software with complete documentation. 

The presentation of this paper will focus on the results 
from these two pilot classes using TSPi and peer 
instruction interventions. Data collected will be 
compared with results collected duringprior studies [1], 
]5]. Due to the new dimensions e^lored in the current 
work, some additional questions have been added to the 
previously used questionnaire. In particular we will 
report on the role of TSP in determining students’ 
assessment of the team experience. Data reported will 
include productivity (lines of code and time spent), as 
well as qualitative data measuring satisfaction with the 
experience and grades received for the software and 
documentation deliverables. 

Peer instruction interventions include having students 
from the prior term conduct the first class of the 
following term, having project teams take “team 
quizzes” during the first two weeks of the term, and 
having student teams review the work of other student 
teams. Students who have been successful in the 
Systems Analysis and Design course show their project 
to the new class and explain what must be done to be 
successful in the course. The students present the 
syllabus as well. This intervoation proved to be highly 
effective. Secondly, quizzes during the team formation 
period are taken on an individual basis and then as a 



team. Individuals receive an average of the two grades 
on the condition that the individual grade is a passing 
score. This activity increased team coherence and 
motivated students to study so that they could contribute 
to the team quiz session. Finally, student teams evaluate 
the systems request, feasibility study, and process 
models of other teams. Students agreed that this review 
helped them to see needed improvements in their own 
work. Furthermore, they found the work of peers more 
interesting than textbook examples. 

In sum this presentation will focus on two new 
dimensions likely to impact the effectiveness of team 
process: software engineering approaches and peer 

instruction interventions. Results achieved along each 
dimension will be reported and compared with prior 
work. Recognizing the importance of teamwork in the 
professional environment, many IS educates require 
that students complete team projects. The overall goal 
of this research is to enrich and deepen the learning that 
occurs when IS students work on these team projects. 
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Extended Abstract 



The problem of bringing students’ programming 
and system development abilities along from 
beginning to pre-professional levels is one which 
confronts most MIS programs. 

In an introductory course, the enphasis tends to be 
on language syntax and individual projects. If 
students later undertake a capstone project, the 
assumption is likely to be that they have previously 
acquired a considerable variety of “hard” and “soft” 
skills. These range from technical issues, like 
database design, to softer — though by no means 
easier — skills, like requirements analysis and 
interface design. 

The road between those levels is a long one. 
Students naturally need to increase their level of 
technical competence. At the same time, they need 
to learn a variety of other skills, many of which are 



more readily acquired in a practical setting than in 
a lecture context. Some examples include: the 
ability to interact with non-technical users, to work 
with incomplete or ambiguous program descrip- 
tions, to divide work among team members, and to 
design and manage projects so that their fit and 
finish can be adjusted based on project progress and 
schedule imperatives. 

This forum is intended to offer participants an 
opportunity to share experiences and ideas about 
how such “intermediate” projects can be structured 
and managed to work more effectively for students. 
Areas of interest include, but are not limited to: 
topics and sources for such projects, how they 
should be managed, levels of project autonomy, 
types of users and interaction with them, and 
assessment methods. 
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ABSTRACT 

This paper examines issues concerning the design, management, and assessment of team projects in e-commerce 
development courses. The paper is composed of three parts. The first part examines the conceptual foundation in 
e-commerce and a user-centered Web engineering method. This concept and method form the content of an e- 
commerce development course. The second part presents a case study of how team projects can be designed, 
managed, and evaluated in the context of a graduate level e-commerce course. This section provides an in-depth 
analysis of the intent, deliverables, and process for each phase of the team project. Also discussed are insights and 
tactics about how to manage and appraise student teamwork. The third part of the paper presents a set of practical 
strategies. The appendix section includes a comprehensive checklist for assessment and a peer evaluation form for 
project teams. 



INTRODUCTION 

This paper examines issues concerning the design, 
management, and assessment of team projects in e- 
commerce development courses. Team project is an 
essential pedagogy in teaching e-commerce application 
development. This approach emphasizes the integration 
of business concepts and technology (Surendra, 2000). 
Through team projects, students learn a holistic set of 
skills required for Web development (Grundy, 2000). 
By building e-commerce projects and subsequently 
using the projects to simulate online retailing, students 
experience the Web development process as well as the 
operation of an e-commerce site in a virtual economy 
(Dhamija, Heller, and Hoffman, 1999; Rosenbaum, 
2000). Team projects can also be used for teaching the 
concepts and process of Web design, programming, and 
usability in a non-profit context or for community 
service (Lazar, 2000). 

Team projects for systems development typically follow 
the systems development lifecycle (SDLC) and the joint 



application development (JAD) method. However, 
successful adoption of a team project approach for e- 
commerce courses may require different considerations. 
Compared with traditional systems development teams, 
the composition ofan e-commerce project team involves 
a more diverse mix of skills and roles. A collaborative 
process is critical to a team’s success (Burdmann, 1999). 
The dynamic interaction between technology and e- 
commerce strategies dictates a more interactive 
development process. Furthermore, a focus on user 
experience and usability issues introduces new 
requirements for design (Melsen, 2000). These factors 
imply that e-commerce development courses need to 
consider different strategies, methods, and processes for 
team projects. 

In this paper, I discuss the issues and strategies affecting 
the design, management, and assessment of student team 
projects for e-commerce develq^ment. The p^er is 
composed of three parts. The first part examines the 
conceptual foundation in e-commerce and a user- 
centered Web engineering method. This concept and 
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method form the content of an e-commerce development 
course. The second part presents a case study of how 
team projects can be designed, managed, and evaluated 
in the context of a graduate level e-commerce course. 
This section provides an in-depth analysis of the intent, 
deliverables, and process for each phase of the team 
project. Also discussed are insights and tactics about 
how to manage and appraise student teamwork. The 
third part of the paper presents a set of practical 
strategies. The appendix section includes a 
comprehensive checklist for assessment and a peer 
evaluation form for project teams. 

ESSENTIAL E-COMMERCE TOPICS 

Courses for e-commerce development need to cover 
both business concepts and the application of 
technology and tools (Suiendra, 2000). This dual 
emphasis limits the range of business topics that can be 
introduced in such courses. Management oriented 
courses may cover a broad range of e-business strategies 
and models, incorporating even a multi-disciplinary 
emphasis (Schneberger, Parent, and Haggerty, 2000). In 
contrast, topics congruent to the application 
development process facilitate project success. Thus, 
the selection of relevant e-commerce topics for a 
project-oriented course is an important decision. 

A framework or model for online retailing can guide the 
selection and organization of course topics. By now, 
most students have already experienced online 
shopping. It is easier for them to apply online retailing 
models than extranets in team projects. Building B-to-B 
solutions demands advanced knowledge of technology 
(such as XML, Java, and message queuing) and business 
strategies (such as supply chain management and e- 
marketplace). A consumer-centric model is useful, 
because it dissects customer/user behaviors and the 
interaction between the merchant’s Website and its 
customers. Both the e-commerce value chain and the 
mercantile models, discussed below, focus on user 
experience. Topics relevant to these models can form 
the core content of a course. Furthermore, such courses 
should emphasize the shopping cart process and 
usability issues. Inattention to these issues could cause 
failure in online retailing. 

The E-Commerce Value Chain Model 

The e-commerce value chain can serve as a framework 
for organizing course topics. It presents a marketing 



perspective of customer relationship management. In 
this model, an e-commerce Website relates to its 
customers in four stages — attract, interact, act, and react 
(Treese and Stewart, 1998). Through marketing, adver- 
tising, and promotion, a merchant attracts potential 
customers to its Website. Through effective 
presentation of catalogs, products, and user interfaces, 
a Website interacts^'iih. its potential customers, with the 
hope of converting them from shoppers to buyers. 
Using a well-designed shopping cart, a Website acts on 
capturing the transaction. Through customer services, 
post-sale support, and personalization of customer data, 
a W ebsi te reacts to satisfy the customers ’needs in order 
to build a sustained relationship with them. 

Topics relevant to the value chain include business 
models, value propositions, consumer behaviors, 
security, encryption, digital payment, marketing, 
advertising, personalization, and privacy. Introducing 
these topics helps students develop a greater 
understanding about each link and the inter-relationships 
among the links of the value chain. 

The Mercantile Model 

The mercantile model, developed by Kalakota and 
Winston (1997), presents a customer perspective for 
online retailing. This model groups the consumer 
mercantile activities into three stages — pre-purchase 
interaction, purchase consummation, and post-purchase 
interaction. The pre-purchase interaction phase involves 
the search and discovery of products based on attribute 
comparison, pricing, and delivery. The purchase 
consummation phase specifies the flow of information 
and transaction between the customers and the 
merchant, including order placement, authorization of 
payment, and receipt of product. The post-purchase 
interaction phase addresses customer service, support, 
and return policies. 

Compared to the e-commerce value chain model, the 
mercantile model focuses more on the customer’s 
behaviors, but less on the conversion of the relationship 
from one stage to the next. It does not address the 
attract process and how a Website can leverage the 
information gathered from the act and react phases for 
building personalized services and repeat purchases. 
Therefore, one may use this model to supplement the e- 
commerce value chain in order to provide students with 
a fuller understanding about the steps involved in 
product search, comparison, and transaction processing. 
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A Focus on Shopping Cart 

The design of shopping and checkout process merits a 
close look. Shop.org, the online group of the National 
Retail Federation, reports, “approximately 65% of 
shopping carts were abandoned before purchase” (2000). 
This high shopping cart abandonment rate represents 
lost sales. Ernst and Young’s Global Online Retailing 
Report (2001) shows that 78% of shoppers reported they 
had placed products in an online shopping cart, but 
failed to complete the purchase. Among the reasons 
cited are complaints about the long and confusing 
checkout process. Students need to understand that a 
seamless and smooth shopping experience is critical to 
the success of online retailing. 

Usability for E-Commerce 

Usability for e-commerce has attracted increasing 
attention as many companies recognized the high cost of 
doing business on the Web. Even after the downfall of 
dot.coms, there is still fierce competition for impatient 
customers in cyberspace. Several industry oriented 
usability reports have focused on guidelines for e- 
commerce Website design. These reports propose a set 
of design guidelines for the category pages, checkout 
and registration process, product pages, and user’s trust 
(Nielsen, Farrell, Snyder, and Molich, 2000a, 2000b, 
2000c, 2000d). Students can learn the importance of e- 
commerce usability by applying specific guidelines 
when building e-commerce sites. 

USER-CENTERED WEBSITE ENGINEERING 

Methodology for Website development does not yet 
have the maturity of software or system development. 
Among various approaches in practice, a user-centered 
approach to Website engineering seems most prominent 
in supporting e-commerce needs. This approach aug- 
ments a modified waterfall model of SDLC with user- 
centered design. 

Website Engineering 

The modified waterfall model is grounded in software 
engineering and resembles systems development. It 
involves six phases: (a) problem definition and concept 
exploration, (b) requirement analysis specification, (c) 
design prototyping, (d) implementation and unit testing, 
(e) integration and system testing, and (f) operation and 



maintenance. To support Web development, a spiral of 
iterations may be added during the early phases of the 
process (Powell, 1998). Nevertheless, Website 
engineering encompasses at least three interrelated sets 
of design — information design, interface design, and 
program design. Information design deals with the 
organization of a Website’s content. Interface design 
refers to the navigation and user interaction at a 
Website. Program design involves constructing a site’s 
functionality and interfaces with back-end information 
systems. Achieving a balance among information 
presentation, interface, and functionality is the key to a 
successful Website. In contrast, most software and 
system development focuses primarily on functionality. 

User-Centered Design 

Many companies developing e-commerce Websites 
have adopted user-centered design methodologies 
because a Website’s success hinges on its being easy to 
understand and use. User-centered design emphasizes 
usability testing and design guidelines for the design of 
a Website’s information architecture, navigation, 
interfaces, and functionality. By using user/task 
analysis and card sort techniques (Nielsen and Sano, 
1994; Rosefeld and Morville, 1998), a Website can 
better reflect the mental model of its intended users. By 
using paper prototyping and following design 
guidelines, a site’s navigation and interfaces can be 
easier to use. Therefore, incorporating the user focus in 
Website engineering makes the development process 
more productive for Website development. 

Site Usage Analysis 

The competitive nature of e-commerce solutions 
demands the collection and analysis of site usage and 
performance data. Inclusion of topics on the measure- 
ment and usage of such data helps students to 
understand the need for continuous monitoring the 
impact of a Website. Internet technology allows 
tracking of usage data through log files, cookies, and 
user registration. By requiring students to identify 
metrics of usage and business measures, students learn 
to identify how log files can be analyzed for data 
mining, and personalization purpose. They develop a 
better understanding about the advantages and 
limitations of common measures, and about how to tie 
marketing, sales, and promotion together by using 
various site measures. 
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Collaborative Process 

Web development employs a collaborative process 
among team members of diverse skills. An ideal project 
team will include skills in graphic design, copy writing, 
marketing, project management, information 
architecture, usability testing, coding, database, network 
architecture, and business strategies (Burdman, 1999). 
These roles reflect different professional training and 
culture. The fast pace of Web development, the 
aggressive launching timelines, and the iterative design 
process of user-centered Web engineering method 
demand close collaboration among team members. 

CASE STUDY 

This section of the paper discusses how team projects 
can be structured and managed to support a graduate 
level e-commerce development course. I used the case 
study format to analyze the course that I have taught 
over the past few years at DePaul University’s School of 
Computer Science, Telecommunications, and 
Information Systems. The pedagogy for student team 
proj ects in this course has evolved and refined over time 
to incorporate student feedback in project team peer 
evaluation (see Appendix B). Most of student results are 
qualitative and informative (in contrast to summative) in 
nature. 

Course Overview 

The course on e-commerce development is offered over 
eleven weeks in a quarter system. Students need to 
achieve three learning goals: (a) demonstrate an 
understanding of e-commerce models, strategies, and 
value chain with an emphasis on consumer-related 
issues; (b) apply the process of user-centered Website 
engineering method; and (c) integrate business strategies 
and solution development. Through team projects, 
students apply their knowledge in concepts, methods, 
tools, and technologies in building e-commerce 
solutions. Additionally, students have to conduct 
Website analyses, participate in class discussion, and 
complete exams. Each student receives a computer 
account to publish the Web analysis. Members of each 
team share a project account to set up a team Website 
for posting project deliverables, and use the same 
account to publish their e-commerce Web projects. A 
course server is provided, which supports ASP scripting 
and Access database. Students can choose their 
preferred software for development. Most students in 
this class have completed prerequisite courseworic in 



systems analysis and design, HTML, JavaProgramming, 
JavaScript, and ASP programming. Some of them have 
taken courses in human-computer interactions. After 
completing the e-commerce course, they move onto 
advanced courses in usability, supply chain technology, 
data mining, intranet development, and e-business 
strategies. Within this context, the group projects 
provide students a structured process of team learning. 
The following discussion presents an in-depth analysis 
about the intent, deliverables, and the process for each 
phase of the team project. Also discussed are insights 
and tactics about how to manage and assess student 
teamwork. 

Team Projects 

The ultimate goal of team projects is to prepare students 
to work as Web developers in the e-commerce 
environment. Accordingly, students form project teams 
and function as Web consulting groups contracted by 
clients to develop Web solutions. As shown in Table 1, 
each project consists of eight phases. Each phase needs 
to follow prescribed process and produce specific 
deliverables. The end products include the prototype 
version of a fully functioning Website for e-commerce, 
along with complete documentation of each step of the 
development. 

Specifically, each project includes the following 
components: 

• Develop a business case for consumer-oriented 
Internet commerce. The project can be a dot.com 
case, a click-and-moitar case, or a site for non-profit 
organizations. 

• Analyze client requirements, including strategies, 
intended customers, content organization, business 
process, functions, and branding. 

• Provide the Website’s information architecture. 

• Design site layout, content presentation, user 
interfaces, and transaction processing. 

• Implement the proposed Website. Each Website must 
include database-driven functions, such as a shopping 
cart, search functions, and membership registration. 
The completeness of each site will depend on the 
complexity of the proposal and a team’s experience 
and size. 
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• Prepare a feasibility report that includes technical and 
budgetary considerations. 

• Propose evaluation criteria to measure the success of 
the Website. 

Team Formation 

Students form project teams of five or six members 
during the first class. Each team includes members 
performing roles of project lead, analyst, designer, 



media/graphic designer, programmer, database 
developer, information architect, usability consultant, 
and recorder/scribe. Sometimes one of the team 
members also serves as a representative user or client. 
Team members play multiple roles, especially during 
the production phase. Each team posts a team 
organization statement (Deliverable A) on their team 
Website, which lists the names, skills, experience, roles, 
and responsibilities of each team member. They may 
sign up for the Blackboard discussion forum or use 
email as the communication protocol. 



TABLE 1 

PHASES, DELIVERABLES, AND PROCESS FOR TEAM PROJECTS 



Week 


Project Phases 


Deliverables 


Process 


1 


1. Team 
Formation 


A. Team Organization Statement 


Establish communication protocol. 

Sign up for Blackboard discussion group. 
Build a team Website. 


2-3 


2. Business Case 


B. Business Case Statement 


Identify potential clients. 
Develop business cases. 
Perform user and task analysis. 


3-4 


3. Analysis 


C. Requirements and Specifica- 


Conduct card sort (optional). 



5-6 4. Design 



7-8 5. Feasibility and 

Site Evaluation 
and Promotion 



7-11 6. Development 

and Prototyping 
and Testing 



tion Report: information 

architecture chart, transaction 
flowchart or data flow diagram 
D. Design Report: Sketches of 
layout, navigation, and tem- 
plates, ERD, and data tables 



E. Technical and Budget 
Feasibility Report: technical 
solutions, cost projection, and 
metrics for measurement 

E. Evaluation Criteria and 
Promotion Strategies in 
Feasibility RqDort. 

F. Prototype Website: Codes and 
database. 



11 


7. Presentation 


F. 


Presentation of Project and ] 
Demo of the Website. 


11 


8. Documentation 


G. 


Final Documentation Folder ] 
and Project Team Website ( 



Expand requirement details. 

Chart the steps for transaction process. 

Revise information architecture chart. 
Prepare page descriptions. 

Sketch alternative designs and templates. 
Perform data modeling. 

Gather content and graphics. 

Gather cost data and prepare estimates. 
Compare technical solutions and hosting 
options. 

Identify usage metrics and impact measures. 
Identify promotion strategies. 



Reach consensus on templates and layouts. 
Prepare paper prototype for user testing. 
Develop cascading style sheet. 

Code applications. 

Prepare database. 

Test applications and performance. 

Prepare presentation. 

Revise and finalize deliverables in print and 
online version. 
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Team composition is the first issue to address. Small 
size, complementary skills, commitment to a common 
purpose, performance goals, and approach are critical to 
successful teamwork (Katzenbach and Smith, 1993). 
Based on my experience in teaching this course, the 
optimal size for productive teamwork is four or five 
members. Six would be the maximum size. E- 
commerce projects require a broader range of skills 
(Burdman, 1999). A small team usually cannot capture 
skills for usability testing and grapiiic design. There are 
two ways to form teams — self-selectiai or assignment 
by the instructor; each has its pros and cons (Wells, 
2002). In either case, clear requirements regarding the 
roles and tasks to be perform by each team helps 
students to find appropriate partners. Starting the team 
formation at the onset of the course signals the energy 
and the pace required for Web development. Creating 
a team Website he^s project teams build a common 
identity. Some teams have even created sites that 
resemble boutique Web consulting shops, and turn this 
task into an effective vehicle for building teamness. 

Business Case 

Each team has two weeks to identify potential clients 
and explore project ideas. Once the team confirms a 
project idea, they need to analyze the business proposal, 
user profiles, and tasks. They then prepare a business 
case (Deliverable B) to explain the proposed project in 
terms of its business models, value propositions for the 
client and the target customers, competitors in the 
market, the rationale for using the Internet solution, and 
how the Internet channel relates to other established 
channels. 

Creating a high-level business case encourages students 
to identify key issues early. Before students engage in 
the business case, they have to complete an individual 
assignment analyzing a pair of Websites. In this 
assignment, students compare the business models, 
value propositions, target customers, market competition 
of two Websites, and how these two sites implement the 
e-commerce value chain. Even with this preparation, 
students still often have difficulty articulating value 
propositions for customers, but they have an easier time 
defining the benefits for the client organization. The 
instructor should address this issue eariy because a clear 
understanding of the user profile and goals facilitates the 
modeling of the user’s mental model. Gathering 
information from representative users can help teams to 
define their projects with a clear focus on users. 



There are three approaches to the selection of a 
candidate project: (a) working with a real client, (b) 
working with a pseudo client, and (c) creating a 
fictitious client. A real client provides students realistic 
scenarios and a well-defined project scope and content. 
In the pseudo client approach, students can collect 
information from a potential client to build the case but 
don’t have to work directly with the client afterwards. 
If students do not have access to real clients, they can 
build a fictitious business case using existing business 
models, such as a bookstore, a travel agency, or by 
creating an innovative solution. There are pros and cons 
of working with a real client or creating a fictitious 
client, as discussed at the end of the paper. 

Requirement Analysis 

Each team then expands the business case into a 
requirement statement (Deliverable C). In the 
requirement statement, students provide details about 
the proposed projects in terms of measurable goals, the 
tag message for the site, sources and categories of 
content, site structure and interpretation, sensorial 
design considerations, and methods for market testing. 
They also prepare two charts: an information 

architecture chart to show the organization of pages on 
the site; and a flowchart to map out the steps of the 
shopping cart process. 

Several issues often surface during this phase. Students 
tend to create an information architecture chart from the 
developer’s or the organization’s perspective. 
Therefore, the chart reflects database functions and an 
inward view of the site. Sometimes, they confuse the 
logical structure of a site with the site navigation. The 
instructor should encourage students to find representa- 
tive users (both novice and experienced users) to help 
shape the site’s infamation architecture. The card sort 
technique (Nielsen and Sano, 1994; Rosefeld and 
Morville, 1998) can help to capture a user’s mental 
model of the information being placed on a site. A 
complete information architecture chart should include 
all the pages to be produced for the proposed site. Later 
on, the chart is used for estimating the amount of work 
and helping the division of labor during the production 
phase of the project. Some students feel more 
comfortable using the DFDto chart the information flow 
for the shopping cart. However, a detailed flowchart is 
more suitable because of the document-centric nature of 
Websites. A flowchart, presenting each step of a 
shopping cart (order creation, order capture, payment 
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authorization, order fulfillment, and order confirmation) 
or registration, facilitates the programming task. 

Detailed Design 

If the information architecture chart and the flowchart 
for transaction processing are complete, a team can 
proceed to develop detailed descriptions of each page on 
the information architecture chart. The graphic designer 
should provide sketches of site layout, color choices, 
navigation systems, and templates. The database 
designer should perform data modeling andprepare data 
tables. The result of these tasks constitutes Deliverable 
D. Revisions for the information architecture chart and 
the flowchart are submitted along with Deliverable D. 

A recurring observation during the design phase is 
students’ reluctance to prepare sketches and paper 
prototypes. Instead of a pencil-and-paper version, many 
students, using Photoshop or one of the graphic tools, 
are inclined to produce a lab version of the layout and 
templates as the first draft. This makes revision time- 
consuming, and often discourages creativity. The 
instructor should encourage teams to present alternative 
designs. A team may not agree on a template until 
several iterations of redesign in sketches. The instructor 
should formally approve the final version of the 
template before allowing the team to proceed with 
programming. Otherwise, different team members may 
work off competing templates or argue about layout and 
colors during the production phase. 

Technical and Budget Feasibility 

At this stage of the project, each team has clear ideas 
about the scope of work and technical solutions to 
prepare a feasibility report (Deliverable E). The feasi- 
bility report addresses the requirements for software, 
hardware, and the hosting service. A proforma budget 
includes the costs for technical solutions and labor for 
site development and maintenance. Each team needs to 
research the costs for the proposed solutions and scope. 
The report should also include strategies for site 
promotion and evaluation, and at least five criteria for 
measuring the site’s impact and usage. 

Textbooks on systems analysis and design always place 
the feasibility study early in the development process. 
I have found that without a thorough understanding of 
the Web development process and technology 
requirement, students cannot produce a meaningful 
feasibility report. Only at this stage of project 



development, can students grasp the project scope and 
the necessity for the mix of talents required for 
implementation. The purpose of a feasibility study is to 
help students see the financial and technical 
requirements for implementing the proposed project. 
Therefore, the completeness of the items identified in 
the report is far more important than the accuracy of the 
estimates. The inclusion of site metrics helps students 
to explore the potential use of such data for site 
maintenance and improvement. 

Development and Testing 

Before entering the programming phase, students can 
negotiate with the instructor to re-scope the project. 
Once the scope is set, each team engages in building the 
prototype e-commerce site as specified in Deliverables 
C and D. Development and testing take place during the 
last four weeks of the term, so the project lead has to 
coordinate the team work with vigilance. A cascading 
style sheet needs to be developed first to allow several 
programmers to work concurrently with a consistent 
look and feel for the site. Database programmers have 
to build and test the ASP functions. The usability 
consultant has to test the consistency of navigation and 
site performance across different browsers and 
bandwidths. 

Several problems occur in this phase. In spite of the 
emphasis on the shopping cart process, students tend to 
ignore the design guidelines and details in programming. 
Edit masks and feedback are not consistently provided. 
Search functions do not follow usability principles. To 
ensure project success, I had to set very specific criteria 
for the shopping cart process, the registration process, 
search functions, cookies, cascading style sheets, and 
ASP scripts (see Appendix A). To make testing 
efficient, students only need to test their sites using the 
Microsoft IE and Netsc^e browser, but they do have to 
specify the browser setting, version, and display setting 
for testing. Version control is a challenge for group 
work. Under time pressure, team members may work 
off different versions of site, thus wasting valuable time. 
Students have reported “it was challenging to write code 
without version control. Our group wasted precious 
time trying to figure out what code was correct or 
incorrect.” Advanced development tools, such as Visual 
Studio, support teamwork through version control. 
However, most basic tools do not support version 
control. A project lead would have to set clear 
communication rules to avoid collusion among multiple 
developers. 
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Presentation 

The presentation at the end of the class gives each team 
an opportunity to demonstrate their Website and share 
their development experience with the rest of the class 
(Deliverable F). Each team member is required to 
participate. The presentation highlights the information 
in Deliverables B through E. The focus is on the project 
Website itself. 

The final class is often filled with excitement. Students 
enjoy the intense experience and the interactive process 
of Web development. Many students are surprised at 
how much teamwork can lead to very professional 
Websites. For the presentaticxi, project teams should be 
encouraged to use their team Websites (for deliverables) 
and the project Websites as the basis to present a story 
of their business case. These two Websitesprovide rich 
information for an interactive presentation. They should 
be discouraged from producing separate Power Point 
presentations. 

Documentation 

Each team presents the project documentation in two 
forms— the online version and the print version 
(Deliverable G). Students can make necessary revisions 
on any deliverables. 

Documentation is an important element of team 
projects. Students should develop good habits early. An 
online version of the project deliverables expires after 
the class is over. The print version provides students 
with a good material for their job search. In both 
versions of the project documentation, each team 
includes the first draft and revised copies of information 
architecture charts, the flowchart, and the site templates 
to document the iterations that took place. 

Assessment 

The assessment of e-commerce Web development 
projects needs to consider several factors. First, the 
assessment should encourage iterative designs. Instead 
of grading each deliverable throughout the term, I only 
provide feedback to teams during the process, and grade 
the project as a whole at the end of the term. This 



grading method motivates students to explore different 
options and collaborate. Second, the assessment should 
balance the quality of the products (deliverables and the 
final project) and the quality of the process. Active 
participation in the process of e-commerce Web 
development leads to effective Websites. As shown in 
Table 1, a clear definition of the process leading to each 
deliverable is important. Third, the assessment should 
reward individual and team contribution separately. 
Assigning one grade to everyone in the team is unfair to 
those who have made extra commitment; it also breeds 
the passive participation of individuals who fail to carry 
their share of responsibilities. 

I have developed two instruments for assessment. 
Appendix A includes a comprehensive checklist to guide 
the grading of team projects at the end of the term. This 
checklist includes 65 items that correspond with the 
process and product aspects of each deliverable. It is 
particularly useful for guiding the prototype develop- 
ment because many design guidelines are built into the 
evaluation criteria. Appendix B includes a peer 
evaluation form that every team member fills out at the 
end of the term. The evaluation form asks students to 
rate their own performance and contribution. They also 
evaluate their peers by distributing 100 points among 
teammates. In addition, each project deliverable 
specifies contributions made by each team member. 
Therefore, project grades are assigned based on these 
three sources of data. 

Over the years, I have adjusted the weight assigned to 
individuals and to the team (see Table 2). Now 
individual contribution accounts for 30%, instead of 
1 5%, of a student’s project grade. Using data of project 
grades of 24 project teams (122 students) over four 
terms, there is a noticeable change in how peer 
evaluation is reflective of individual students’ effort. 
When individual contribution accounted for 15% of the 
project grade, only 9.4% of students (among 64 students 
or 13 project teams) received less than full grades for 
team contribution. When the weight of individual 
contribution was raised to 30%, 34.5% of students 
(among 58 students or 1 1 project teams) failed to attain 
the full score for individual contribution. 
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TABLE 2 

EVALUATION CRITERIA FOR TEAM AND INDIVIDUAL PERFORMANCE 



Team Performance (70%) 


Individual Performance (30%) 


Team effectiveness 


Timely attendance at meetings 


Clarity of project scope and objectives 


Timely completion of assigned tasks 


Completeness of analysis 


Useful contribution to projects 


Consistency and creativity in design 


Supporting other team members 


Sound technical and budget solutions 


A fair share of responsibilities 


Working prototype system 


Extra share of responsibilities 


Effective presentation 


Peers’ allocation of 100 points 


Completeness and quality of documentation 


Documented contribution 



Student Feedback 

According to student response to the peer evaluation 
form (in AppendixB), team projects in this course have 
provided students unique insights about e-commerce 
Website development. They learned a great deal about 
“working with people of varying degrees of skill and 
experience/’ “virtual collaboration methods due to 
differing schedules and availability,” and “the challenge 
associated with developing a user-centered Website.” 
One student stated “the project provided me with insight 
into the entire Website design process, something that 
was new for me. It was interesting and informational to 
invoke the waterfall method in this capacity and watch 
the site developed accordingly.” Many students 
acknowledged, “the developmait of an e-commerce 
Website is very extensive and requires the cooperation 
from many different fields.” Furthermore, “proper 
planning and design are important before starting a task. 
It is better to spend more time on planning than making 
changes later.” Among factors contributing to a 
project’s success, most often cited are teamwork, 
communication, a broad mix of skills, and effective 
project management. Teamwork involves “a good share 
of responsibilities,** **a willingness to help each other,” 
“ability to work both as individuals and groups,” and 
**everyone finished his/her responsibilities on time and 
was able to attend every team meeting.” Effective 
communication involves “constant communication via 
email, phone, and meetings,” and an **open-mindedness 
shared amongst the team which allowed each member to 
learn something new.” Students also indicated that their 
ability to **put to the best uses of the broad mix oftalents 
and skills on various parts of the project ** was critical. 
Students emphasized the importance of project planning 
and having **extensive to-do lists” to ensure **continued 
communication with the team on issues and deadlines.” 
The aggressive project scheduledictates “everyone is up 



to date with his/her responsibilities,” and effective 
project leadership can **make or break” a team. As to 
team leadership, ability to **assign people based on their 
talents or skills can get the job done quickly and have 
better results.” 

PRACTICAL STRATEGIES 
Working with Real Clients 

Working with real clients is well worth the time and 
effort involved. Students gain valuable experience in 
learning how to work within constraints to meet the 
business needs of the client. Websites designed for real 
clients often reflect a well-defined scope and overall 
design. However, real clients may expect different 
outcomes from what the instructor expects. If their 
expectation is too high, students may not be able to 
complete the project within the allotted course time. 
More often, real clients are uncertain about going online 
and hence only want a brochure ware site. In the first 
case, student teams diould build a prototype project to 
satisfy the course requirement and complete additional 
enhancement after the course is over. In the second 
case, the instructor’s expectation defines the complexity 
of the project. Students need to present the proposed 
site as a prototype to illustrate how an interactive online 
retailing site may help the client to achieve its long-term 
goals. Working with non-profit organizations presents 
different challenges from working with commercial or 
corporate clients. Students learn to focus on novice 
users as well as organizations with less technological 
savvy. Design and usability decisions have to sustain 
waves of progression as the organization and its staff 
goes through their learning curves. Nevertheless, 
students should always seek q)portunities to work with 
real clients for Website development. 
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Web for Project Management 

A team project Website is an effective tool for project 
management and team communication. Having a central 
place to publish all the project deliverables makes it 
easier for team members to achieve a common 
performance goal. The interactive nature of hypertext 
documents allows a site to link all deliverables and 
supporting documents easily to show the progress of 
iterative designs. Writing project deliverables in 
hypertext documents is a good exercise for students to 
practice active and succinct presentation. Students gain 
experience in learning how to link texts with graphics, 
charts, references, and the project site to form a 
convincing story of their e-commerce project. Because 
the brevity and the multi-media potential for a team 
Website, students can explore creative ways to use the 
site for project management. For teams that have to 
collaborate across distance, the team Website facilitates 
virtual project management. 

A Gallery of Projects 

The instructor can set up a gallery Website to exhibit 
exemplary team projects. The gallery page should 
include the name of all the team members, their project 
deliverables, and the project site. Students not only 
enjoy the public recognition of their work, but can also 
direct potential employers to the URL’s as part of their 
online portfolio. The gallery site also serves as a 
valuable instructional resource. However, when real 
clients launch their official sites, the prototype site 
developed for the course should be removed to avoid 
confusion. Otherwise, the search engines’ robots may 
index both sites and lead real customers to the wrong 
sites. 

Managing Team Progress with Assessment Tools 

A clear definition of the structure and process for team 
projects increases the chance of the projects’ success. 
An assessment tool, like the checklist shown in 
Appendix A, makes students attentive to the process of 
e-commerce Web development. Assessment criteria for 
the production and programming phase are especially 
useful. Instead of expecting students to incorporate 
appropriate design guidelines during programming, 
detailed assessment criteria remind students to 
implement easy-to-use interfaces and functions. 



Instructors should discuss the assessment tool at the 

beginning of the project to set proper expectations. 

Therefore, the assessment tool is an effective means for 

managing student teams. 
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APPENDIX A 

A CHECKLIST FOR EVALUATION OF TEAM PROJECTS 

1. Team Effectiveness (10 points) 

□ 1.1 The project team has a clear definition of roles and responsibilities for eadi team member. 

□ 1.2 Team members have complementary skill sets and experiences. 

□ 1.3 The Project Lead exercises effective leadership in conflict resolution and project management. 

□ 1.4 There is an even distribution of responsibilities among team members throughout the project. 

□ 1.5 The team has regular and open communication and records resolutions for each meeting. 

□ 1.6 The team posts project deliverables in a timely manner on the team’s Website. 

2. Clarity of Project Scope and Objectives (10 points) 

□ 2.1 The project has appropriate business models and measurable goals. 

□ 2.2 The project demonstrates clear value propositions for the crganization. 

□ 2.3 The project demonstrates clear value propositions for the users and customers. 

□ 2.4 The project provides a clear profile of intended users. 

□ 2.5 The project provides an analysis of market competition. 

□ 2.6 The project has a reasonable scope for implementation. 

□ 2.7 The project demonstrates clear justification for an Internet solution. 

□ 2.8 The project provides a profile of the client organization. 

3. Completeness of Analysis (15 points) 

□ 3.1 The information architecture supports the business goals and value propositions. 

□ 3.2 The site’s structure supports the goals and tasks for both novice and experienced users. 

□ 3.3 Each page on the site is clearly defined and represented in the information architecture chart. 
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□ 3.4 Details of the site’s transaction process and interactive processes, such as shopping cart and registration, 

are clearly mapped out in flowcharts and/or Data Flow Diagram. 

□ 3.5 The project provides appropriate sensorial design requirements. 

□ 3.6 The project identifies technical and organization constraints for implementation. 

□ 3.7 The information architecture chart reflects user task analysis. 

□ 3.8 The information architecture chart incorporates results from card sort with users. 

4. Consistency and Creativity in Design (20 points) 

□ 4.1 The project provides an ERD model and data definition for the site. 

□ 4.2 A sketch of site template and page layout is developed first before implementation 

□ 4.3 The team explores alternative design templates and design ccxicepts. 

□ 4.4 The site uses consistent templates, interfaces, and navigation placement. 

□ 4.5 The site’s navigation follows usability guidelines. 

□ 4.6 Users can access the site’s main pages and functions within three clicks. 

□ 4.7 The site uses special interfaces to personalize feedback, services, or content presentation. 

□ 4.8 The site uses effective graphics (low download time) and colors (browser safe colors). 

□ 4.9 The site uses creative design to enhance branding and usability. 

□ 4.10 The project team performs a paper prototype before implementation. 

5. Sound Technical and Budget Solutions (10 points) 

□ 5.1 The technical solutions for this site provide appropriate recommendations for hardware, software, and 

hosting services. 

□ 5.2 The budget solutions provide detailed costs and resources for implementation and maintenance. 

□ 5.3 There are clear criteria for measuring the project’s success. 

□ 5.4 The project team identifies appropriate site usage metrics. 

6. Prototype Site (20 points) 

□ 6.1 Users can view the shopping cart at any time during the shopping process 

□ 6.2 Users can remove and update projects in the shopping cart. 

□ 6.3 The shopping cart is emptied once checkout is done. 

□ 6.4 All the product information is populated from database. 

□ 6.5 Users can continue shopping before checkout without losing the content of the shopping cart. 

□ 6.6 User registration supports novice and experienced usa*s. 

□ 6.7 Novice users can register at the checkout point during the first time shopping at this site. 

□ 6.8 Order confirmation provides personalized online feedback. 

□ 6.9 Order confirmation provides e-mail notification. 

□ 6.10 Search facilities match the content and the format of data returned. 

□ 6.1 1 Advanced search facilities provide instructions and examples. 

□ 6.12 Negative search results pages include information on why a query failed 

□ 6.13 All the data entry functions provide edit mask and feedback. 

□ 6. 14 All the hyperlinks work properly. 

□ 6.15 All the ASP scripts function properly. 

□ 6.16 All the sound/video features and plug-ins work properly. 

□ 6.17 There are no typos in content. 

□ 6.18 The prototype project uses cascading style sheets for consistent presentation. 

□ 6.19 All the CSS rules work correctly and consistently in the IE and Netscape browser. 

□ 6.20 The prototype site uses cookies to track repeat users. 

□ 6.21 The prototype site includes a privacy policy about the user of cookies and personal information. 



7.0 Presentation and Demo (10 points) 

□ 7.1 The prototype site works successfully with all data input and feedback. 

□ 7.2 Every tern member participates in the presentation. 
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n 7.3 The presentation provides a clear business case and site design/development process, 
n 7.4 The presentation materials are well organized. 

8.0 Completeness and Quality of Project Documentation (5 points) 

n 8.1 The project deliverable site is well managed, complete, and easy to navigate, 
n 8.2 The final project folder includes the final version of all the deliverables. 

n 8.3 The final project folder also includes the original copy of informationarchitecture chart and screen layout 

(sketches). 

n 8.4 The final deliverables lists the tools and technology uses for development and testing. 



APPENDIX B 

PROJECT TEAM PEER EVALUATION 

List each member’s first name. Fill in the number that best describes each team member’s contributioa 
1 = rarely met expectation; 2 = met expectation; 3 = exceeded expectation 





Self 










1. Attended project meetings on time. 












2. Completed assigned tasks on time. 












3. Made useful contribution to the project. 












4. Supported other team members. 












5. Took a fair share of responsibilities. 












6. Took an extra share of responsibilities. 













1. List your contribution to the team projects in terms of specific tasks and deliverables. 

2. How would you distribute 100 points among members of your team (excluding yourself) on the basis 
of each person’s contribution to the project? 

3. What factors have contributed to your team’s success? 

4. What lessons have you learned from this project experience? 
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Colleges of business are particularly known for assigning large-scale team projects as a means of enhancing 
learning. Though students may assemble in one location for ^^traditional*' classroom-based learning, time and 
distance may create challenges as students attemptto fulfill requirements associated with large-scale team projects. 
This study explores the use of collaborative group tools found in common course management systems (e.g., 
Blackboarcf and WebCT^) and community sites (e.g., MSN Communities^) as a means to address logistic and project 
material organization problems associated with large scale team projects in colleges of business. We investigate 
relationships among social and technical factors to enhance the learning outcomes that can result from student 
project teams deploying collaborative group tools. Specifically, this study uses qualitative inquiry of team 
stakeholders to compare perceptions of use of collaborative group tools from both a user and non-user perspective 
and develop a causal model identifying potential net benefits from the use of collaborative tools and antecedents to 
benefits to facilitate current understanding as well as future research. Additionally, this study provides prescriptions 
for use of collaborative group tools to facilitate successful application of collaborative group tools to student group 
projects. 



INTRODUCTION 

If only it was as easy as a simple math problem: 

Addend 1-Team projects in business courses 
+ Addend 2-Team time, logistic, and project 
organization challenges 

+ Addend 3-Team access to virtual group 

collaborative technologies 

= Net Gains (Well, maybe!) 

Globalization, the distribution of team members, 
technology support, and a quest for efficiency, has 
propelled the existence of both successful and 
unsuccessful virtual and semi-virtual team structures in 



professional organizations and a resulting viable 
research stream. In a similar fashion, technological 
advances (and access), the distribution of members, and 
a quest for efficiency, are propelling a new context for 
semi-virtual or virtual teams in an academic setting, 
namely the business student group project. 

Colleges of business are particularly known for 
assigning large-scale projects (extending the duration of 
a semester or multiple semesters) as a means of 
enhancing learning. Students may congregate in one 
place to attend class sessions; however, logistical and 
schedule barriers may impede project team assembly 
outside of the classroom meeting time. The barriers may 
be exacerbated in situations where a team project is 
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assigned in a distance-learning context or where a 
project connects distributed team members from 
different universities or geographic locales. As such, the 
use of collaborative technologies to support student 
group projects provides a situation of virtual or semi- 
virtual teams existence as well as a means to further 
explore technology-mediated learning (particularly in 
the development of computer mediated communication 
skills in the context of teamwork). 

Common course management tools like Blackboarcf 
(http://www.blackboard.com/) and WebCT® (http:// 
www.webct.com/) allow instructors/course managers to 
create group sections on the course web site to facilitate 
project work. The features of such group sections 
typically include threaded asynchronous communication 
(e.g., group discussion board), synchronous communi- 
cation (e.g., group virtual classroom), file storage, and 
group e-mail capabilities. Additionally, publicly 
available sites, such as MSN Communities® (http:// 
communities.msn.com/) can provide students with web 
site space providing community message board, 
calendar, link, photo albums, lists (e.g., to do list), and 
custom web pages in addition to all of the features 
associated with the course management tools. 

When faced with a team project in a distance education 
course or in a classroom based course, instructors may 
make these collaborative technologies (hereafter, project 
team tools will be referred to as collaborative techno- 
logies) known and available to students or students may 
independently seek such collaborative technologies. 
Collaborative technologies may be used as an alternative 
to some or all face-to-face meetings, to facilitate 
communication among team members, and/or as a 
means to organize course project materials. 

RESEARCH OBJECTIVES 

These collaborative technologies hold the promise of 
facilitating what seems to be an immediate need for 
project teams in colleges of business - communication 
convenience, member access, and project organization. 
As well, exposure to collaborative technologies may 
prepare students for virtual team arrangements they may 
face in the future in the work place. However, it seems 
little is known regarding situational propriety and the 
antecedent factors to fulfilling technological promises 
(Izard & Reeve, 1986; Patel & Russell, 1999). In a 
recent article addressing the status of technology 
mediated learning research in IS, Alavi and Leidner 
highlight a void in research studies that “focus on 



forming relationships among technology and relevant 
instructional, psychological, and environmental factors 
that will enhance learning outcomes’’ (Alavi & Leidner, 
2001). In response to this void, we seek to investigate 
relationships among social and technical factors to 
enhance the learning outcomes that can result from 
student project teams deploying collaborative 
technologies in colleges of business. 

Specifically, this research seeks to satisfy three 
obj ecti ves to enhance understanding of this phenomenon 
and facilitate successful application of team 
collaborative technologies to student group projects. 

• Compare perceptions of use from both a user and non- 
user perspective. 

• Develop a causal model identifying potential net 
benefits and their antecedents to facilitate current 
understanding as well as future research. 

• Provide prescriptions for use based upon qualitative 
inquiry of team stakeholders. 

Our basic research question can be stated as, “What 
social and technical factors are necessary to make web- 
based collaborative technologies a viable resource to 
students in project teams?” 

To attend to these objectives, we provide the theoretical 
and logical foundation serving as an underpinning to 
field inquiry, present the interpretive fieldwork 
performed in this study and the emerging model, and 
provide insight to guide research and practice. 

THEORETICAL FRAMEWORK 

Contextual Domain 

Researchers engaged in the study of IS success urge the 
academic community to pay careful attention to domain 
context in defining and measuring each component of 
success included in studies (DeLone & McLean, 2002; 
Seddon, 1997). Research also indicates that users prefer 
to tailor constructs and measures to the type of system 
under evaluation to facilitate understanding and 
application (Jiang & Klein, 1999). Though research in 
virtual teams in a professional context may provide 
some insight, it is our position that the deployment of 
collaborative technologies in education is a context with 
innate idiosyncrasies that merits distinct research to 
develop a comprehensive model that identifies 
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situational relevant variables and relationships. 
Professional projects staffed by distributed teams exist 
in a multitude of forms; conversely, there is general 
consistency in form that distinguishes the academic 
project deploying the collaborative technologies. 
Traditional idiosyncrasies in the student team project 
context follow. 

• Academic project teams are short lived (typically not 
exceeding a semester) and have inflexible deadlines. 

• Team members do not have predefined roles in 
entering the project. 

• Team members may not have history with or even 
knowledge of other members. 

• Team assignment is an informal and perhaps even 
arbitrary process. 

• Outside of perhaps class time, team members do not 
have a predefined work location and hours for team 
activities. 

Furthermore, it is difficult to equate the motivators (e.g., 
the motivation of grade pressure) of student team 
members to the motivators (e.g., the sustenance and 
embitterment of a livelihood) of professional team 
members. 

In response to these unique characteristics, we seek to 
explore the phenomena of interest from a ground-up, 
context specific perspective using an interpretive 
paradigm recognizing the socio-technological nature of 
the phenomenon. 

Socio-technical System 

The deployment of collaborative technologies tosupport 
a distributed (or semi-distributed) work arrangement is 
undeniably a socio-technical system as the system is a 
composition of technology, human interaction, and the 
environment. As such, we must look beyond the 
technology in attending to the purpose of this study. 

Computer mediated communication technologies 
(CMC) have been employed facilitating distance 
communication in synchronous, asynchronous, and 
multi-mediaforms. CMC technologies — electronic mail, 
computer conferencing, distribution lists, and chat 
rooms — have all been employed to enable and support 
distributed work arrangements. However, ambiguity 



regarding whether these technologies create richer and 
more meaningful communication exist (Kraut, Rice et 
al., 1998). Kraut refers to this ambiguity as the 
computer-mediated communication paradox. The basic 
assumption of CMC theories is that CMC does not 
transmit the same level of vibrancy and interaction that 
exists in face-to-face inta*action (Lisetti & Bianchi, 
2002). Therefore, users of CMC systems are thought to 
adapt to their CMC technology and therefore to exhibit 
fewer of their natural communication behaviors. 

These findings render it increasingly important for 
researchers interested in virtual collaborations and 
collaborative technologies to tdkQdi socio-technological 
approach to their studies. This provides guidance to 
process as well as product as a product’s success could 
be enhanced by developing technology that adapts (or 
co-adapts) to social processes (such as a medical visit) 
at the center ofhuman lives (Lisetti & Bianchi, 2002). 

User Based View 

To address the socio-technical perspective adequately, 
we adopt a user-based view. It is important to 
understand what users perceive to be important, because 
it is these issues that underpin their perceptions of 
information system success (Whyte, Bytheway et al., 
1 997). According to the user-based approach, end users 
are the ultimate judge andjury regarding the technology. 
As stated by Whyte, the difficulty with the user-based 
approach is to find the set of variables that strongly and 
positively correlate with the perceived performance of 
successful information systems. To deal with this 
difficulty, we query users and non-users as respondents 
to identify the issues, variables, and relationships 
associated with the deployment of collaborative 
technologies in this academic context. 

Voluntary Use 

We restrict the scope of this study to the use of team 
collaborative technologies as a supplement to a 
“traditional,” classroom based business course to 
support project teams (hence, the possibility of semi- 
virtual teams). We specifically sought contexts of 
voluntary use during the course of this study where 
students plausibly have the option to meet face-to-face 
during the course term to work on project activities. We 
select this context in order to expire both use and non- 
use given otherwise equal parameters (e.g., potential 
users enrolled in the same course). We feel voluntary 
deployment would provide the greatest insight into the 
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research objectives as indicated by prior IS research on 
technology acceptance (Venkatesh & Davis, 2000). 

METHODOLOGY 

Procedures 

We used open-ended survey instruments as a means of 
qualitative inquiry to explore student perceptions and 
experiences with using collaborative technologies to 
support group projects. Open-ended survey instruments 
versus structured interviews were used to allow 
respondents to reflect on their response, provide 
information at a convenient time, and to access the 
collaborative tool with project team content for 
reference purposes. Two forms of survey instruments 
were used — one for students that opted to use 
collaborative technologies and one for students that did 
not. 

Items on the two instruments were the same or closely 
parallel where possible to facilitate comparison between 
the use group and the non-use group. Survey instrument 
items were derived through analysis of transcripts 
related to five semi-structured interviews with 
instructors requiring a major course project that enable 
group features in their course web system (e.g.. 
Blackboard® or WebCT®) or present the use of publicly 
available tools like MSN Communities® and the 
experiences of one of the authors in using these tools in 
classroom settings. The interviews began with a digest 
of the research intent. There were two basic questions 
guiding the interview process. 

• What questions would you recommend asking 
students to learn more about the application of these 
tools to their course projects? 

• What do you believe are some of the high-level issues 
in using these tools? How would you recommend 
inquiry into these issues? 

These questions sparked further interview probes and 
discourse. Four instructors (two of whom had 
participated in the interview process) servedas apre-test 
panel to review open-ended questions developed. Open- 
ended questions were piloted using five students. Pilot 
and pre-testing resulted in minor revision to items and 
instructions. Open-ended questions included in the 
instruments are provided in Appendix A. 



Context 

Given our objective of explorative inquiry, we sought a 
context which would provide a breath of use situations 
(courses, instructors, and collaborative technologies) to 
enhance the potential for comprehensive stu<fy. We 
chose to conduct this study within a College of Business 
at a large metropolitan university in the southeastern 
United States. This site was conducive to the study as 
the university academic computing system supported: 

• multiple forms of course environments (Blackboard® 
and WebCT®), through technology, training, and 
“customer support,” for at least three years prior to the 
study (utilization of these tools was at the discretion of 
the professors), 

• instructors in various departments who had chosen to 
use various tools, 

• free dial-in Internet access for students, and 

• many on-campus computer labs for students. 

We spoke with instructors from various departments to 
identify those courses that included a large-scale 
(semester duration) group project and those instructors 
who apprised students of and afforded students the 
opportunity to use collabc^ative technologies in some 
form. We specifically sought situations where the 
instructor had taught the identified course before and 
had repeatedly used the same form of group project. We 
primarily addressed undergraduate courses as an attempt 
to focus our study to users/non-users not already using 
parallel tools in a woric setting. However, we did collect 
information from a team of graduate IS students 
working on a project for comparative purposes. 

Research Process 

We contacted instructors teaching relevant courses 
before the start of a semester requesting participation. 
Instructors teaching screened courses and willing to 
participate were identified in the Information Systems, 
Finance, and Marketing Departments. Students in four 
upper-division undergraduate courses participated in the 
study along with one graduate project team. Each of the 
instructors confirmed they apprised students of 
collaborative technologies available throughout the 
course as an elective means to facilitate group projects. 
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At the close of the semester, instructors discerned which 
student teams used collaborative technologies and which 
did not and proceeded with data collection. Information 
regarding the study was not provided to the students 
prior to this point in time. After announcing the topic of 
study, students were asked to complete the appropriate 
survey instrument. The instructor distributed respective 
surveys and related instructional matmals to students. 
Students were directed by their instructors and survey 
instructions to return the surveys to the researchers via 
campus mail in the provided self-addressed envelope. 
Assurance of anonymity was conveyed both through 
verbal instructions from participating instructors and 
through the survey instructions. The response rate was 
92% (113 of 123 surveys distributed were returned). 
Table 1 provides the distribution among disciplines and 
user versus non-user. Table 2 provides a brief 
participant history broken down by user/ non-user. 

Responses to open-ended questions were analyzed to 
discern antecedent and consequence factors of effective 



use by one researcher using an inductive method for 
developing a coding schone described by Boyatzis 
(1998). A second researcher analyzed the responses 
using the developed coding scheme noting suggestions 
for refinement/ improvement to the schema. Two 
additional researchers apprised of the study reviewed the 
coding schema and coding details. An iterative process 
of review betweai the researchers continued until 
thematic convergence and an agreement on labels and 
definitions was reached. The model presented in Figure 
1 resulted from this iterative process among researchers. 

The final model is represented in Figure 1. Each of the 
constructs is supported by a preponderance of evidence 
resulting from this analysis process of respondent 
comments. Once the model was finalized, responses 
were then reanalyzed by two researdiers to determine if 
statements supporting the construct were from users, 
non-users, or both (see Table 3). 



TABLE 1 

PARTICIPANTS BY DISQPLINE AND USE/ NON-USE 



Discipline 


Collaborative 
Tool Users 


% Collaborative 
Tool Users 


Collaborative 
Tool Non-Users 


% Collaborative 
Tool Non-Users 


Total 


Information Systems 


14 


56 


11 


44 


25 


Finance 


0 


0 


16 


100 


16 


Marketing 


18 


24 


56 


76 


74 


Total 


32 


27 


83 


73 


113 



TABLE 2 

PARTICIPANT HISTORY— USER/NON-USER 





Users 


Non-Users 


Total 




History Item 


Yes 

(%) 


No 

(%) 


Yes 

(%) 


No 

(%) 


Yes 

(%) 


No 

(%) 


Did you participate in group projects in college of 
business courses prior to this course? 


13 

(41) 


19 

(59) 


43 

(53) 


38 

(47) 


56 

(50) 


57 

(50) 


Have you used course web sites in the past? 


26 

(81) 


6 

(19) 


56 

(69) 


25 

(31) 


82 

(72) 


31 

(27) 


Have you used group features of course wd) sites 
or web communities in the past? 


4* 

(36) 


1* 

(64) 


17* 

(37) 


31* 

(65) 


21* 

(36) 


38* 

(64) 



*This question was added to the survey after 54 surveys had already been collected. 
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FIGURE 1 

COLLABORATIVE TECHNOLOGIES TO SUPPORT STUDENT TEAM PROJECT MODEL 
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TABLE 3 

DISTRIBUTION OF CONSTRUCTS INDICATED 
BY USERS AND NON-USERS 



Model Constructs 


Users 


Non-Users 


Technology 


Organized efficient 


X 


X 


repository 






Reliable 


X 


X 


Basic/Simple Design 


X 




Fast and Easy to Learn 


X 


X 


Communication Options 


X 


X 


Team (Critical Mass) 


Participation 




X 


Convenience 




X 


Individual Assessment 


Learning Curve Investment 


X 


X 


Proximal Course Value 


X 


X 


Ancillary Value 


X 


X 


Informed and Effective Use 


X 




Communication Gains 


X 




Organized Materials 


X 





DISCUSSION 

Based upon quantitative data analysis, informed and 
efficient use depends upon three dimensions of issues: the 
technology itself, individual assessment, and team critical 
mass. This section defines each of the model constructs 
associated with these dimensions, provides a trail of 
evidence via quotes and paraphrases from respondents, 
and addresses differences between users and non-users. 

Antecedents to Informed and Effective Use — 
Technology Related Constructs 

Organized, efficient repository. Both users and non- 
users asserted that the collaborative technologies should 
provide effective file management features. Respondents 
noted collaborative technologies should provide a/an 
“place to organize one’s work,” efficient (storage) 
system,” and “place to organize and store materials.” 

Reliable. Reliability encompasses failures of the system 
as well as problems encountered in accessing the system 
through log in. Both users and non-users expressed the 
need for dependable technology. There seems to be 
reliability issues with some existing systems in use as 
users noted that the ^^site was not always working'' and 
referred to mechanical failures of the site, " 



Basic/simple design. Interestingly, only users expressed 
the need for a basic and simple design. Perhaps non-users 
did not spend enough time in exploration to make an 
evaluation or design issues were best discovered through 
the course of use over the semester. Design complexity 
issues were particularly noted in user responses to 
collaborative technologies disadvantages through com- 
ments such as, it was “confusing about where to post 
messages,” and the site was “difficult to understand.” It 
seems that designers of collaborative technologies 
specifically designed for educational use should be 
judicious in adding features as students noted there were 
“too many tools to use.” 

Fast and easy to learn. To facilitate informed and 
effective use, the technology must accommodate an 
efficient learning process. Both users and non-users 
indicated that the collaborative technologies introduced 
initial learning challenges through such remarks as, the 
technology is “too hard” and “takes too long to learn to 
use.” Comments indicate that initial challenges during the 
learning process dissuaded further use for the project. 

Communication options. Communication options refers 
to the availability of various features provided by the 
technology to aid communication and discussion among 
group members as well as supporting technology 
infrastructure to provide access from any location at any 
time. As stated by one respondent, the technology 
provided “easy communication among group members at 
all times.” Regarding communication feature set, users 
indicated the desirability of discussion board, meeting 
scheduler/calendar, and e-mail. Conversely, virtual 
classroom, chat rooms, and ironically e-mail were stated 
as being the least useful features in the sites. 

Antecedents to Informed and Effective Use — 
Individual Assessment Constructs 

The deployment of these technologies involves an 
individual assessment as well as the collective assessment 
of group use. We will first address the individual 
assessment constructs. 

Learning curve investment. Users reported an average 
time required to feel comfortable in using the collabora- 
tive technologies of 36 minutes. On the surface, this 
would not seem to be a substantial learning curve for a 
new software tool. However, one user noted a “long 
learning curve” was an issue. It seems most non-users 
may have assumed a substantial learning commitment 
was required without actually investing the time as 
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represented by a failure to report an exploratory time 
investment and comments such as it is ‘‘too time 
consuming to learn a new technology” and there is “not 
enough free time” to learn a new technology. 
Communication from instructors informing students of 
learning curve estimates may allow students to perform 
a more accurate individual assessment of time 
requirements. 

Proximal course value. From a negative perspective 
regarding the project itself, some students noted the use 
of the collaborative technologies to be “information 
overload” or “unnecessary.” As would be expected in a 
decision process of voluntary use, students assess 
whether the technology will add additional value to 
project outcome. Additionally, students noted 
collaborative technologies can actually enhance 
pedagogy as represented by the comment that the 
collaborative technologies “facilitated class learning.” 

Ancillary value. Ancillary value in terms of personal 
enjoyment was expressed in reference to “love of 
computers.” Additionally, a positive assessment 
regarding the use of the tools may encompass an 
expectation of use beyond the course. Both users and 
non-users indicated that collaborative technologies 
should be of beneficial use for future courses and even 
future work. Though there may be justified reasons 
some colleges allow choice in the course site 
technologies (e.g.. Blackboard® versus WebCT®), 
comments regarding ancillary value for future courses 
imply the use of collaborative technologies within a 
specified institution would benefit from homogeneity. 
Respondents indicated the potential of ancillaiy value 
regarding a work setting through the comments related 
to “beneficial for all types of work,” “useful to know for 
business and industry,” “helpful in all facets of future 
jobs,” which seemed to facilitate a positive assessment 
and use. 

The innate nature of some disciplines, such as 
information systems, may indirectly convey the 
utilization of technology through the educational process 
will benefit future work settings. Individual instructors 
may inform the individual assessment process by 
communicating potential future value related to future 
courses and work settings. 

Antecedents to Informed and Effective Use — Team 
(Critical Mass) Constructs 

The collective assessment of the group does not seem to 
require an absolute positive individual assessment by all 



group members. However, it seems critical mass is 
necessary if collaborative technologies are to be used 
effectively. 

Participation. Students identified challenges faced in 
using the collaborative technologies when a critical 
mass of group members do not employ the technology 
or reluctantly use the technology. As stated by one 
respondent, I “cannot use site because not all group 
members use (the site),” Statements by users indicate 
a lack of collective participation can definitely cause a 
group to abandon use of the collaborative technologies. 

Convenience. Collectively, the group needs to 
determine the collaborative technologies provide a 
convenience that will address situational barriers. In a 
semi-virtual team situation, face-to-face and telephone 
meetings may be alternative means of communication. 
Group communication dynamics (and skills), geographic 
dispersion and schedule issues will influence perceived 
communication conveniences among choices. Situa- 
tional factors indicate collaborative technologies do not 
provide a convenience when members find “other 
methods of communication easier” and “prefer meeting 
in person.” 

Consequents to Informed and Effective Use 

Organized materials. Users stated collaborative tech- 
nologies provide a means to organize materials used, 
safeguard files, and increase the overall organization of 
the project. In describing the value of collaborative 
technologies, one student statedit “provided information 
pertaining to the project at all times.” 

Communication gains. Communication gains can be 
thought of as enhancements to the communication 
process that would not be realized barring the use of the 
collaborative technologies. In communicating the 
communication values of collaborative tool use, users 
noted the technology both facilitated “ease of 
communication” and afforded “multiple forms of 
communication.” 

General Reactions 

Though the aforementioned discussion provides 
indications that non-users reported barriers to using the 
web site, the majority (68%) indicated they would use 
the collaborative technologies in the future. As such, 
there does seem to be some recognized value though a 
technology may not be utilized for a particular course 
project or by a particular group. 
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Users also indicated they experienced some issues 
during the deployment process. However, a holistic 
review of user comments indicates they found the use of 
collaborative technologies to be positive. Furthermore, 
users indicated (97%) that they would use the 
collaborative technologies to facilitate future projects. 

CONCLUSION 

Technology is increasingly sought as a means to address 
distance, scheduling, and student volume in higher 
education as well as a means to complement classroom 
learning. If we are to understand and realize the 
potential of such technologies in education from a 
learning and access perspective, we mustunderstand the 
perceptions of both users and non-users as well as 
recognize the nuances of the educational context in 
deploying tools that may also find use in professional 
contexts. From a research perspective, this study 
acknowledges these issues in explaining and modeling 
a particular existing phenomenon which has received 
little attention in literature, but seems to be growing in 
use, namely the availability of collaborative tech- 
nologies to support group projects in college of business 
courses. Further work may explore the peculiarities of 
group dynamics especially in the initial stages of group 
formation that may affect the successful use of these 
tools, which was not particularly addressed by this 
study. 

The results of this study provide prescriptive insight for 
various stakeholders in practice. For the designers of 
web-based technologies for classroom use, Bauhaus 
philosophies seem to hold, namely less is more in the 
eyes of the user. The short-lived nature of the project 
and the team may not provide the conditions where 
extensive functionality can be fully exploited and may 
even deter utilization. For college computing support 
infrastructures, the perceived ancillary advantages of 
using the same collaborative technologies/on-line course 
system in all courses deploying systems merit 
consideration. Likewise, there appears to be an apparent 
need for the reliability of collaborative technologies and 
implicitly course web sites that may be accessed by 
students 24/7. Additionally, computing support and/or 
instructors may want to consider providing some basic 
“training” on using collaborative technologies that 
includes coverage of virtual communication skills. 

Finally, instructors should recognize that like many 
professionals, students may also face distance and 



scheduling issues which complicate the execution of 
group project tasks. The availability and information 
regarding the use of collaborative technologies may 
provide a viable option to mitigate issues for some 
project teams and could enhance the learning process 
intended by the project assignment. 
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APPENDIX A 

OPEN-ENDED SURVEY QUESTIONS 



User Questions 

Questions were customized to the course project and the collaborative technologies made available to facilitate 

question clarity. 

1. What did you like (advantages) about using the name of collaborative technologies for the name of project 
assignment ? 

2. What did you dislike (disadvantages) about using the name of collaborative technologies for the name of project 
assignment and course ? 

3. Approximately how long didit take before you became comfortable in using name of collaborative technolog ies? 

4. What facilitated your learning process and comfort level in using name of collaborative technologies ? 

5. What difficulties did you encounter in using Azame of collaborative technolog ies? 

6. What hampered your learning process and comfort level in using name of collaborative technolog ies? 

7. Explain any effects on team dynamics you noted by using nam e of collaborative technologies (positive and/or 
negative). 

8. What features did you find most useful? 

9. What features did you find least useful? 

10. What changes in use would you recommend to the design or method of using name of collaborative 
technolog ies? 

1 1 . How do you feel using a name of collaborative technologies to facilitate a business course project differed from 
your academic project experiences that did not use a community site? 

12. What information did you add/contribute to name of collaborative technolog ies? 

13. Would you use a community site in the future? If so, in what context? 

Non- User Questions 

1 . What did you like (advantages) about the idea of using the name of collaborative technologies made available 
associated with the name of collaborative technologies made available for the name of project assignment and 
course ? 

2. What did you not like (disadvantages) about the idea of using the name of collaborative technologies made 
available associated with the name of collaborative technologies made available for the name of project 
assignment and course ? 
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3. Did you try to use name of collaborative technologies for the project assignment and course? If so, 
approximately how much time (hours and minutes) did you spend in exploring the name of collaborative 
technolog ies! 

4. What discouraged you from using name of collaborative technologies for the name of project assignment and 
course ! 

5. Did you find the group feature set in name o f collaborative technologies was lacking? If so, what additional 
features would you have liked (for example, file directory/subdirectory structure, assignment list capabilities, 
group calendar, etc.)? 

6. How did your team members react to using the name of collaborative tool (positive and/or negative)? 

7. Do you think the name of collaborative technolog ies have any value for student group projects? Please explain 
your response. 

8. Would changes would your require to use name of collaborative technologies in the future (e.g., demonstration, 
instruction sheet)? 

9. Would you use name of collaborative technologies in the future? If so, in what context? 
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SUMMARY 



This article proposes a categorization model for online distance education environments, based on two different 
aspects: interaction and content. The proposed categorization, which was based on the experience acquired in 
developing, implementing, and operating different remote training courses, is aimed at providing evidence to help 
educational professionals in choosing online environments that are more appropriate to their educational goals and 
strategies. 



INTRODUCTION 

Web-based online education environments are now 
becoming very important educational tools [Schrum, 
2000 and Parrel, 1999]. These online environments can 
be educational software packages available on the 
market, such as Lotus Learning Space [IBM 
Corporation, 2000] , WebCT [WebCT Inc, 2002], or 
experimental products developed by Universities, such 
as AulaNet [AulaNet Project, 2002] and WebCurso 
[Reinhard, 2000]. In addition to these online education 
environments, collaboration environments, such as 
eRoom [eRoom.Net, 2002], have also been used as 
virtual tools to support remote education. 

Most of the Internet-based virtual environments that can 
be applied to remote education were developed through 
the integration of synchronous and asynchronous 
communication tools, such as chat, discussion forums 
and lists, and electronic mail. These projects range from 



developing specific tools that are fully integrated with 
the virtual education environment to using separate tools 
to create an on-line environment. WebCurso [Reinhard, 
2000], Virtual-U [Harasim, 1999] and Lotus Learning 
Space [IBM Corporation, 2002] are examples of 
environments that are based on the integration of their 
own communication tools and the environment. 
Microsoft, for instance, adopts the separate tool- 
approach, which, although does not offer virtual 
application environment ^ecifically directed towards 
remote education, provides all the required tools to 
create such environment. 

These environments can include several tools and 
different fQSituvQS. Appendix 1 - Functionalities of Web- 
based online education environments presents a 
(incomplete) list of features that can be fully or partially 
found in most of the environments. Our proposal, which 
is presented below, states that these functionalities or 
features actually define how the virtual environments 
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can be used within the remote education and training 
context. 

This article presents a categorization for distance 
education environments, based on the educational 
purpose and two different dimensions: — interaction and 
content. The proposed categorization was based on the 
experience acquired in developing, implementing and 
operating different distance education environments and 
training courses that will be used as examples. The 
proposed categorization is aimed at helping education 
professionalsto select the most effective environment to 
meet their teaching-learning goals and develop 
educational strategies. 

OTHER AUTHORS’ 
PERSPECTIVES ON THE TOPIC 

According to Sherry [1996], several researchers assign 
the same meaning, both to remote education and remote 
training, based on the idea that the main component is 
the distance — both in terms of space and time — between 
the teacher and the student and the teacher-student 
communication process is mediated by technology. 
Other writers, such as Valente [2002], assign different 
meanings to “remote training” and “remote education,” 
that is, “training” means to provide information, and 
“education” means to capture information and allow 
knowledge building. 

Harasim [1 990] summarizes the characteristics of online 
courses as (1) time and place independence; (2) peer-to- 
peer communication, and (3) dependence on text-based 
communication to stimulate learning through under- 
standing and reflection. The main strengths of such 
environments would be using synchronous and/or 
asynchronous communication, reaching a wide range of 
students regardless their location, and creating learning 
communities. 

However, there are innumerous examples in the 
literature showing that the benefits of online courses are 
not easily readiable, considering that: 

• Online courses are effective mainly for motivated 
students [Schrum, 1998] or previously educated 
students (able to process information) [Valente, 2002]; 

• Developing educational material for online courses 
can be a tremendous challenge, given the fundamental 
role it plays in the learning process [Schrum, 1998]; 
and 



• Creating learning communities is a complex task, and 
the integration among students is always less intensive 
than expected [Wiesenberg and Hutton, 1996 and 
Reinhard, 2000]. 

Reid and Woolf [1997] have already discussed the 
different features of the online environments, such as 
access capability, student control, increased 
communication and the potential for creating a student- 
centered environment. Heeren and Lewis [1997] suggest 
that special attention should be paid to selecting the 
appropriate feature to each specific educational task, 
matching the media with the task. 

Therefore, we are led to discuss environment-related 
features and technologies and their relation to 
educational activities. Estmond [1998] proposes three 
different types or sets of technologies that can have 
diverse impacts on the teaching-learning process: 

• Type 1 — Traditional remote education supple- 
mented by Internet-based activities. The education 
process is based on printed material delivered to the 
students, self-study, and the Internet is used as a 
supportive resource, which includes e^nails, chats, 
conferences. As the Internet is not fundamental to the 
process, technology knowledge is not extremely 
important. 

• Type 2 — Computer-based conference. In this 
category, the Internet plays the main role in the 
communication and education processes. Although 
printed material can be used, the emphasis is on 
computer-based communication resources, such as 
chats, forums and boards, providing the students with 
synchronous and asynchronous communication 
opportunities. Written communication skills are 
important for the process, as well as technical 
knowledge of computer and Web resources. This type 
of technology requires great interaction among 
students and between the student and the teacher. 

• Type 3 — Virtual courses. This is an extension of 
Type 2, but the Internet plays the role of unique 
source of training material. Type 3 technology-based 
courses, which involve deep immersion in computer 
and Web resources, can provide constructionist 
learning and create learning communities. Remote 
interaction with the teacher (and the classmates) helps 
to provide alternative views and understanding of the 
learning process. 
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Another categorization proposed by Valente [1999 and 
2002] is based on different distance education 
pedagogical approaches, which are defined by the 
different interaction modes provided by the environment 
and adopted by the educational professional: 

• Broadcast. Technology is used to provide information 
to the learners. The main characteristic is that, instead 
of interacting with the teacher, the student interacts 
with the material (ccxitent) prepared by the teacher. 
The teacher receives no (or little) feedback from the 
student, and has no idea on how the information has 
been interpreted or processed by the student. The cost 
of this category per student is extremely low; 
therefore, it is highly efficient for spreading 
information to a great number of students. The main 
role of the teacher is to prepare the educational 
material (or content). 

• Traditional school virtualization. Technology is 
used to reproduce educational activities adopted by 
traditional classroom education. Here the main 
characteristic is the teacher-centered nature of the 
activities; the teacher possesses the information and 
provides it to the students. The interaction between the 
teacher and the student is intensive, allowing the 
teacher to deeply monitor the student’s progress. This 
kind of interaction significantly reduces the number of 
students involved and increases the teacher’s 
activities. The main role of the teacher is to create and 
deliver the educational material to the students as well 
as to provide feedback on the students' activities. 

• VirtuaH^get together.” Technology isused to present 
problem situations or projects to the students. The 
students try to solve the problems individually or 
within the group, and count on the teacher’s coaching. 
Internet-based interaction aims at accomplishing 
“learning cycles” [Valente, 1999], which maintain the 
students involved with innovative activities, allowing 
knowledge generation. This approach was also called 
“learning network” [Harasim, et al, 1 995]. According 
to Valente [2002], thisapproach is extremely complex 
because it implies high costs — the number of students 
is limited and the approach requires a supportive team 
of teachers — and, most important, it demands deep 
changes in the educational processes. The main role of 
the teacher is to create educational material, support 
the learners, and create and maintain a learning 
environment that is appropriate to knowledge 
building. 



Considering the level of cooperation among students 
and between students and teachers, Roberts, Romm and 
Jones [2000] proposed a different categorization model 
to the Internet-based collaborative learning. For this 
purpose, they defined two levels of activities that can be 
accomplished within the online environment: the “group 
level” involves both the teacher and his/her students — 
and everybody participates in the activities and can 
benefit from the work developed by other classmates — 
and the “sub-group level,” which includes small groups 
of student within the same class. These two kinds of 
activities define four different categories: 

ILLUSTRATION 1 
CATEGORIZATION MODEL 
FOR THE INTERNET-BASED 
COLLABORATIVE LEARNING 

Level of Sub-Group Work 



< 



O 
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c 

"U 



€ 
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Online standard 


Radical 


Traditional 


Naive 



low high 



• Traditional. It is originated from traditional classes, 
that is, text material and explanations are used to 
transmit the content to the students. In the virtual 
world, the three main characteristics of this model are: 
the content is delivered through files and text, there is 
no interaction among students, and the interaction 
between the students and the teachers is usually 
through e-mails. The students have no (or few) 
opportunities to learn from their classmates, and the 
learning process strongjy depends on the teacher’s 
knowledge and skills. 

• Naive. The group is divided into sub-groups with 3 or 
4 members; the sub-groups operate as individual units, 
and are independent from other sub-groups. This 
method reduces the large groups of students so they 
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can effectively interact with the teacher; therefore, it 
is often adopted for practical reasons. 

• Online standard. It is based on the workgroup model, 
rather than the sub-group approach. All the students in 
the group are involved in the activities: discussions, 
projects, chats, etc. Therefore, the students have the 
opportunity to learn both JB*om the teacher and their 
classmates. During the development of the activities, 
the students tend to form sub-groups spontaneously. 

• Radical. While the “online standard” and “naive” 
categories can use either the group or the sub-group 
model or both, in the “radical” approach, groups and 
sub-groups are always involved in M educational 
activities. The activities are accomplished at sub- 
group level, based on the interacticxi among the sub- 
group members, but are also developed at group level 
The approach also includes activities developed 
among different groups. This category has very 
distinct characteristics: 

• Mandatory use of environment lists and/or e-mail as 
the only communication alternative available; 



considers the potential of the Web technology to create 
rich education material and classifies the environments 
based on two different dimensions: content and 
interaction: 

• “Content” is herein defined as the subset of 
educational materials available to the students. This 
dimension is justified by the large range of resources 
provided by Web technology. These resources would 
allow the development of virtual educational contents 
that go far beyond the traditional textbooks. The 
content can be presented in different digital media 
formats, such as hypertext documents, streaming 
audio and video, animations, simulations, improving 
the materials with interactivity between students and 
the environment 



poor 



Static HTML 
Plain Text 



PDF document 
MS Word document 



Content 



rich 



Streaming video 
Streaming audio 
Java Applets 
Flash Animation 
Audio and video files 
Video Conrferencing 
VR 




• Online presentations prepared by the students and 
available within the environment; 

• The teacher divides the students in sub-groups, and 
all course tasks are accomplished by the sub-groups; 

• Evaluation is based on sub-group presentations and 
student activities. 



• “Interaction” is the mutual relationship among the 
students (peer-to-peer) and between students and 
teacher. Interaction can be herein understood as the 
frequency, as well as the volume of information 
exchanged among the students and/or between 
students and teacher. 



OUR CATEGORIZATION PROPOSAL 

Although distinct, the three previously described 
approaches have one main common aspect: the 
interaction among students and between teacher and 
students. Also, they all state that to create a distance 
education environment to involve and stimulate the 
students requires intensive involvement of the teacher. 
Individual attention from the teacher is also required 
through private messages and continuous involvement 
within discussion group environments, in order to 
coordinate and direct the discussions toward the course 
goals. 

We herein propose a different categorization model for 
Internet-based distance education environments. In 
addition to the interaction aspects, the proposed model 




Discussion list 
MuK level Forum 
Personal Tutoring 
Group col lab oral! on 
Syncronous conrferencing 
Telepresence 

Therefore, based on the two previously mentioned 
dimensions — interaction and content — and adopting two 
basic measures — low and high (Illustration 2) — -we herein 
propose a distance education virtual environment 
classification based on the interaction level among the 
participants and the richness of the content available to 
them. 
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ILLUSTRATION 2 
DISTANCE EDUCATION VIRTUAL 
ENVIRONMENT CLASSIFICATION MODEL 



Quadrantlll 
Virtual T utoring 


Quadrant IV 
Workgroup 
and/or 

collaboration 


Quadrant 1 
Information 


Quadrantll 

Self-instruction 



poor content rich 



Quadrant I — Information (Low Interaction and 
Poor Content) 

This category is characterized by the availability of 
simple text or files (.doc or .pdf). There is no interaction 
among the participants or the interaction is limited to 
common resources such as FAQs. This category 
includes remote training initiatives or standard informa- 
tion delivery. The student is fully responsible for the 
learning process. 

As an example of this category, we implemented a 
Public Policies (http://www.msco.com.br/cpp) web site. 
The environment aimed at providing information on 
decentralized and participatory management to the 
members of Municipal Councils. The content includes 
only Word text files (or .pdf files) that can be 
downloaded to the student’s computer. The content was 
a manual on the mles that regulate decentralized 
management. The interaction level is low and 
exclusively based on FAQs. The environment is a 
delivery tool to provide specific information on a 
specific topic. 

Quadrant II — Self-Instruction (Low Interaction and 
Rich Content) 

It is characterized by rich and diverse content in 
different formats, but low interaction among the 
students or between the students and the teacher. It is 
appropriate to learning strategies aimed at providing the 



student with a rich virtual environment that allows 
him/her to learn by him/herself, guiding him/her through 
a previously defined sequence of actions. This kind of 
environment is appropriate to large group training 
initiatives, with fixed content and low feedback levels. 

As an example of this category, we implemented two 
different projects: Empreendedor Alerta and 3Com 
Learning Center. Both offer rich content using different 
communication tools. 

The Empreendedor Alerta project is aimed at offering to 
small business owners different contents that teach them 
how to overcome eventual financial difficulties. The 
environment offers Adobe PDF manuals and Microsoft 
.asf streaming video files {Advanced Systems Format^ov 
Windows Media Technology) — see Illustration 3. The 
participant also receives a course kit by mail, including 
a course manual and videotape. 

ILLUSTRATION 3 
DISTANCE EDUCATION 
ENVIRONMENT— EMPREENDEDOR ALERTA 
http://www.msco.com.br/empreendedor 
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The environment counts on limited interactive 
supportive tools — a discussion forum and a specificarea 
for receiving the student messages directed to the 
instructor or to the supporting team. In spite of 
environment limitations, 9,293 people registered for the 
course as of April 2001. Concerning the interaction 
among the students, only 57 message topics and 94 
message answers have been included in the Discussion 
Forum. Considering both the interactions between 
students and the teacher team, 324 messages have been 
registered. These figures show low interaction between 
the students and the teacher team and also among the 
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students — we can notice low interaction among the 
students and between students and teachers, that is, 0.05 
messages per participant. 

Another example of a distance education virtual 
environment that would also be included in Quadrant II 
is the 3Com Learning Center Brazil Project (figure 5). 
The environment is mainly aimed at offering basic 
training on network technology; this is, teaching the 
basics of network tedinology. The environment is an 
adaptation of the equivalent program cun*ently available 
at the 3Com University -USA web site. The environment 
includes 15 different courses on computer networks. The 
exhibition mode is standardized, and includes few 
different,but rich, content formats, using hypertexts and 
animations developed in Macromedia Flash. The 
interaction between the student and the environment 
consists of sequential navigation through the course 
HTML pages, enriched by Flash animations. 

ILLUSTRATION 4 

DISTANCE EDUCATION ENVIRONMENT— 
3COM LEARNING CENTER— BRAZIL 
http://www.msco.com.br/3com 
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The only interaction available is provided by a 
sophisticated FAQ system, that is, the answers to the 
proposed questions can go far beyond the traditional 
FAQs, and can be enriched by file attadimentes and 
illustrations. The student can also directly contact 
3Com: by hotline or e-mail, althou^ no contact is 
provided through the educational environment 

The 3Com Learning Center Brazil is operating since 
June 2001. In April 2002, the number of enrolled 
students was 5.335. Over this period, only 35 student 



messages have been sent to the teacher team. The 
teacher team has sent 6 messages to the students. A 
series of questions presented at the end of each course 
allows the students to evaluate the courses. Course 
evaluation grades range from 1 (poor) to 5 (excellent), 
and are used to measure the students' perception of the 
course. Based on this evaluation tool, we know that 
33.8% of the students consider the courses to be 
excellent, and 46.92% think the courses are good! Only 
16.54% of the students think the courses are regular, and 
less than 3% classify them as poor. These figures show 
that student satisfaction levels concerning this kind of 
environment are high. 

Quadrant III — Virtual Tutoring (High Interaction 
and Low Content) 

The virtual environments included in this category are 
characterized by intensive coaching activities and some 
group work. Considering that the focus is on the 
interaction between the students and the teacher, the 
content does not have to be highly sophisticated in terms 
of audio and video resources. The focus is on tutoring 
actions, that is, providing the student with effective 
support during the learning process. Interaction becomes 
a critical success factor. Communication tools are 
intensively used by teachers, pedagogical coordinators, 
or facilitators to reinforce the learning process. Group 
activities can be adopted to increase the interaction and 
information exchange among the participants. Feedback 
is frequent and highly important in this kind of 
environment, requiring specific tools to monitor course 
activities and student progress. Small and mid-size 
groups— 40 to 80 people (considering only one 
teacher) — can be appropriate to work within this kind of 
environment, considering that interaction between the 
students and the teachers tends to be high. 

As an example of this kind of virtual environment, we 
can mention two remote education projects, 
VirtualCurso and Virtual -U. VirtualCurso (VC) is the 
most current implementation of WebCurso [Reinhard, 
2000]. The environment offers several synchronous and 
asynchronous communication tools to be specifically 
applied to remote education. The environment can be 
configured according to the strategies applied by a 
specific teacher to a specific course. 

From April to September 2001 a remote education 
program was offered to Mathematics teachers (at 
Primary and High School), mainly aimed at training 
them in using spreadsheets to teach Math. The course 
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content, which included a spreadsheet tutorial program 
specially aimed at developing the proposed content, was 
presented through Microsoft Word text files and HTML 
hypertext files, using ”gif’ pictures and screenshots 
from the spreadsheet application. The main strategy 
consisted of: 

• Proposing basic and individual tasks or activities, in 
which the teacher and/or the student would develop a 
series of activities using calculation spreadsheets (in 
this specific situation, Microsoft Excel) and present 
them to the pedagogical coordinator; and 

• Proposing tasks, wich include how to create 
spreadsheet activities related to everyday life, and 
follow the same principles and contents used to guide 
the proposed activities. These files were stored in a 
transfer area (called the “Group Window”), creating a 
database that could be accessed by all the students. 

• Helping to develop the interaction between the 
coordinator and the students. After the evaluation, the 
students were stimulated to improve their spreadsheets 
thorougt the addition of new data and the discovery of 
different alternatives to correct eventual mistakes. 

The chosen strategy significantly increased the number 
of interactions among the students and between the 
students and the pedagogical coordinator. The course 
started with 120 students, was later reduced to 72 
students, and ended with 51 students. Most of the 
interactions were made through e-mail and discussion 
forums, both directly integrated to the VC environment. 
The students sent 1 .523 messages to the coordinator and 
590 to other students. The coordinator sent approxi- 
mately 2.500 messages to the student. Considering that 
5 1 students concluded the course, the average number of 
messages was as follows: 29 student-to-coordinator 
messages, 12 student- to-other-student messages, and 49 
coordinator-to-student messages. The Forum included 
101 topics and 1328 answers to the proposed topics, that 
is, approximately 13 answers per topic. 

At the end of the course, answering the course 
evaluation questions, the students said that they had 
learned a lot, including mathematical, pedagogical and 
computer-related content. All of the 51 students that 
concluded the course answered the evaluation questions. 
Forty-nine of them, (98%), said that participating in a 
virtual environment-based course was a pleasant 
experience, and they would participate again in a remote 
training course. 



Quadrant IV — Collaboration (High Interaction 
and Rich Content) 

This kind of environment is based on peer-to^eer 
collaboration, teacher-student cooperation, and work- 
group activities. Differently from quadrants I, II and III, 
which are aimed at a more individual learning process, 
quadrant IV is based on collaboration activities that 
support the group learning process. This situation 
requires a high level of interaction among the partici- 
pants. Synchronous communication tools, such as audio 
and videoconference, are emphasized in this kind of 
environment, as well as online document sharing (e.g. 
shared white board). Asynchronous communication 
tools, such as discussion fomms and lists, or even other 
tools that allow sending and receiving files (file 
download and upload area) to common areas are 
important within this context. This kind of environment 
allows the students to freely create groups and sub- 
groups. There is a wide variety of contents available 
because, in addition to the material provided by the 
teacher, the students also develop and provide course 
materials. 

Java Collaborative Virtual Workplace (CVW) and 
PauliWorld [Su, 2001] are examples of collaborative 
virtual environments. CVW is used for distributed 
system analysis and distributed collaboration and offers 
a set of resources that allow synchronous and 
asynchronous collaboration through text mode-based 
chats, audio and videoconference, and shared 
whiteboard. According to Maybury [Maybury, 2001], 
CVW implements session persistence, which allows 
recording user interactions as soon as they occur within 
the “shared virtual room” context. 

VirtualTeam also belongs to this category. This 
environment was created and developed aiming at 
facilitating the virtual integration among groups of 
students involved in traditional classroom courses. 
Although the environment does not include synchronous 
communication tools, such as audio and video- 
conference, to deliver sophisticated content, it offers a 
large set of tools to facilitate virtual interaction and 
communication. Its main characteristic is allow sub- 
groups while maintaining privacy. The material shared 
among different groups is based on a resource called 
“window.” Operating since August 200 1 , it is currently 
being used by over 150 students registered in Internet 
Technology MBA courses at Faculdade deEconomiae 
Administra 9 ao, USP. The tool can be seen at 
http://www.virtualteam.com.br. 
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DISCUSSIONS AND CONCLUSIONS 

Before presenting our conclusions, we think it is 
important to reiterate Valente’s statement [2002], that 
the categorization approach is not the only alternative 
available and does not represent preferred technologies. 
The division into categories is just a useful resource to 
help educators choose the most effective environment, 
according to their educational purposes, their target 
audience (number and characteristics), and the related 
social and economical circumstances. 

Concerning the categorization herein proposed, the 
following topics present indications for each of the 
presented categories (see Table 1). 

It is not simple to classify the environments only on a 
single quadrant. The environments may cross quadrants 
boundaries. Illustration 5 shows some of the virtual 
environments mentioned in this paper. 



Each environment includes several features, and a 
specific feature can be implemented in different ways 
within different environments. For example, a FAQ 
system can vary from a simple list of answers in text 
format to a sophisticated multimedia tool associated to 
complex searching mechanisms. 

All the examples presented to illustrate the different 
categories herein mentioned can be considered to be 
successful distance education applications. Although 
completely different, each one is appropriate for the 
intended educational strategy. Therefore, the educators 
are responsible for examining the characteristics of 
different educational environments and finding the most 
adequateto theirpedagogical purposes, reach, resources 
and cost per student. 



TABLE 1 



Category 


Indication 


Example of use 


Information 


Large groups. Explain the new administrative 

Simple and specific topics. Requirement of low cost per student procedures of a company. 
Appropriated to “continuous delivery.” 

Content could be updated periodically. 


Self- 

Instruction 


Large groups. 

More complex topics. 

Low interaction between the students and the teacher. 
Adequated to “continuous delivery.” 

Indicated for information courses. 


Training on computer network 
basic concepts, or training on 
use of an administrative 
information system. 


Virtual 

Tutoring 


Small size groups. 

Focus on the teaching process. 

Interaction between the student and the teacher is the most 
important element. 

Indicated for development courses. 


Teaching the use of Web on 
education and creation 
educational material for use 
on classroom 


Workgroup or 
Collaboration 


Small/Medium size groups. 

Focus on peer-to-peer collaboration. 

Adequate to knowledge exploration activities. 
Constant information exchange is required. 


Teaching how to create and 
develop a business plan. 
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ILLUSTRATION 5 
ENVIRONMENT 
CATEGORIZATION EXAMPLE 
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APPENDIX A 

FUNCTIONALITIES OF WEB-BASED ONLINE EDUCATION ENVIRONMENTS 



Resource 

Manual 

Course material 
Chat 

InstantMessaging 

Forum 

Management Tools 
News 

Files (Database) 
Message box (e-mail) 
Task Management 
FAQ - Technical 

FAQ - Content 
On-line support 
Statistics 

Resource repository 

Evaluation of 
Participants 

Bulletin Board 
Links 

Latest Events 



Description 

Enviro nmen t User M anual in tex t format o r “contex tual help” format. 

Area for providing material to the students. Course material includes text, software, animations, 
e-books, etc. 

Synchronous communication (text-based) for groups, with choice of persistence. 

Synchronous communication (one to one), private, sensitive to presence within the system 
Moderated or edited asynchronous communication. 

Allow creating and monitoring groups and sub-groups, with controlled privacy to be managed 
by teachers. 

News publishing system to be used by teachers/tutors. 

File storage system, including uploadand download, also allowing the creation o f private folders. 
E-mail internal system for internal and external use. 

Assignment and control o f student tasks. 

Frequently Asked Questions sub-system on environment technical issues, which shall serve as 
off-line technical support environment. 

Frequently Asked Questions sub-system on pedagogical issues related to the modules. 

Through Instant Messaging, both text and voice. 

Statistics sub-system on environment usage by stud ents. 

Includes all the required plug-ins so the students can use the resources. 

the Evaluation sub-system that implements “Student Activity Control Card,” including Notes, 
Evaluations and Comments. 

Sub-system that allows both searching information published by participants, and entering 
information to be provided to o ther participants. 

Area that allows entering URLs to be used by students, teachers and tutors; also including 
comm ents. 

Environment monitoring system that allows providing updated information to the students, as of 
the latest entries. 
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ABSTRACT 

This paper analyzes and compares telecommuting in the United States and Mexico. The objective of the paper is to 
answer two research questions related to a framework of telecommuting. The research questions assess the 
productivity, performance, satisfaction, and extent of support by organizational policies for telecommuters in the two 
countries. The tests are done on a questionnaire sample of 1 77 employees in the U.S. and Mexico. The statistical 
method is crosstabulation analysis. 

Results show that telecommuters in the U.S. sample report that telecommuting improves their productivity . However, 
telecommuters in the Mexican sample do not consider that telecommuting improves producti vity. On the other three 
questions, the results show no difference between the two samples regarding job performance, job satisfaction, and 
the extent of support by organizational policies. The findings have implications for telecommuting policies of 
corporations and governments in the two nations. 

INTRODUCTION 

This paper presents the results of an investigation on the 
practice of telecommuting among corporate employees, 
focusing on the differences in experiences and percep- 
tions among telecommuters and non-telecommuters in 
Mexico and the United States. Telecommuting, or the 
ability of employees to work at sites other than their 
corporate office, has been an issue of ongoing interest to 
researchers and practitioners since its introduction in 
1976. Many factors have contributed to keeping this 
issue current, as well as controversial. Initially, 
telecommuting was thought of as an alternative to 
reduce traffic congestion and pollution in large cities by 
keeping employees from commuting from their home to 
their offices (Nilles et al., 1976). Later on, organi- 
zations realized its potential for cost reduction — by 



eliminating needs for office space (Apgar, 1998). Some 
also noted productivity increases, due to elimination of 
work distraction and commuting time for employees 
(Apgar, 1998). Highly skilled individuals realized the 
personal gains of engaging in this working arrangement 
(i.e., autonomy, money savings, and reduced stress) and 
some demanded it as a condition for employment. More 
recently, telecom-muting was invoked as the only 
alternative to resume operations for some businesses 
affected by the September 11, 2001, terrorist attack in 
New York City (Kistner, 2001). 

The many benefits of telecommuting have led many 
individuals and organizations to adopt it. However, there 
have also been repots on the problems that this 
alternative work arrangement brings to individuals and 
organizations. These problems include workers’ feelings 
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of isolation by staying away from the office, work- 
family conflicts due to the inability to separate work and 
family issues, and management’s inability to deal with 
the new mechanisms for controlling and evaluating 
employees. Similarly, other reports find evidence of a 
decrease in productivity among telecommuters due to 
their lack of commitment to the organization or to 
increasing levels of stress brought about by working at 
home. Most likely these adverse reports have kept many 
businesses and individuals from adopting 
telecommuting. 

Recent and more rigorous studies have provided 
explanations to alleviate the controversy, and have 
contributed frameworks to promote further analysis, 
evaluation and discussion of the actual outcomes of 
telecommuting. The work of researchers such as 
McCloskey and Igbaria (1998, 2001) and Belanger 
(1998, 1999, 2001), who conducted extensive literature 
reviews and provided research agendas, have set the 
stage for continuing research on this important issue. 

Building on the work initiated by these researchers, this 
paper explores the state of the practice of telecommuting 
and the experiences and perceptions of telecommuters 
and non-telecommuters. The study analyzes two 
samples, one in Mexico and the other in the United 
States, to empirically measure whether or not 
telecommuters in both countries are indeed more 
productive, better performers, and more satisfied with 
their working arrangement. The study also analyzes 
whether or not organizations support teleworking 
arrangements in both countries. 

Knowing the differences in perceptions and experiences 
among telecommuters will help in:l) promoting 
adoption of this working strategy in both countries, 2) 
designing telecommuting policies in global 
organizations with branches in US and Mexico, and 3) 
policy making regarding the IT industry in Mexico and 
among the Hispanic community in the United States. 
For example, Kramer and Dedrick (2000) recommend 
that developing countries such as Brazil, Turkey, India, 
and Mexico should establish a strategy to utilize IT to 
support government policies and development strategies. 

Based on the objectives of the study, a questionnaire 
was developed and tested using a sample of 177 
corporate employees: 1 1 1 in the United States and 66 in 
Mexico. The experiences and perceptions of these 
employees were captured in their answers to 49 survey 



questions. A prel iminary analysis on the data shows that, 
in concert with recent literature, about 20 percent of the 
sampled population actually telecommuted in the U.S. 
Further analysis on the data indicates that: tele- 
commuters perceive themselves as more productive than 
conventional, non-telecommuting workers. In the case 
of Mexico, the number of telecommuters turned out to 
be too high (45 percent of the sample). A more detailed 
analysis showed that unemployment rate in Mexico 
explains part of this finding, since 50 percent of 
Mexican telecommuters sampled do not work for a 
corporation or other institution. 

LITERARTURE REVIEW 

In the almost thirty years in which telecommuting has 
captured people’s attention, the literature has grown 
substantially. Many importantreports on telecommuting 
have been publicized in practitioner-oriented journals 
and books. Although practitioner-oriented literature is 
very useful in exploring the dimensions of telecom- 
muting, it has also stimulated debate and controversy. 

The virtue of practitioner-oriented literature lies in the 
recounting of the experiences of pioneering organi- 
zations that implemented telecommuting initiatives, the 
outlining of its benefits and drawbacks, and success 
factors that these organizations found throughout their 
adoption processes. For example, Apgar (1998) 
discusses the experiences at IBM and AT&T. 
Davenport and Kerin (1998) present the results of a 
survey of Fortune 500 companies describing the benefits 
and losses of telecommuting for organizations. The 
results of these and others studies are summarized in 
Table 1. The telecommuting benefit touted the most 
among organizations is the reduction of real estate cost 
since fewer employees occupy less office space. 
Additional benefits for businesses as well as for 
individuals are higher productivity and work 
satisfaction. 

However, problems with telecommuting have also been 
reported in the research literature. At the organizational 
level, telecommuting leads to weakened organizational 
culture, lowered commitment, and burdens in managing 
and supervising telecommuters. At the individual level, 
studies report that respondents experience isolation and 
work-family conflicts stemming from telecommuting 
initiatives. Table 2 summarizes disadvantages to 
telecommuting. 
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TABLE 1 

BENEFITS OF TELECOMMUTING 



Apgar, 1998 


• Cost reduction (IBM is saving more than 100 million annually through its Mobility 
Initiative) 

• Potential to increase productivity (At IBM 87% of employees in the Mobility Initiative 
believe that productivity and effectiveness have increased) 

• Opportunity to capture government incentives and avoid costly sanctions. (Clean Air 
Act and tax deductions for hone office costs). 

• Employee recruiting and retention 

• Employee efficiency (fewer distractions and less downtime) 

• Increased personal time and control 


Adam and Crossan, 
2001 


Organization level: 

• Reduced overhead (total office space) 

• Rare skills retention 

• Geographical flexibility 

• Motivation and loyalty 




Individual level: 

• 30 to 1 00% increases in productivity (due to reduced commuting stress, flexibility, and 
appreciation of responsibilities) 

• High satisfaction among teleworkers 

• Uninterrupted concentration 

• Freedom from time constraints 

• Lower living costs: reduced travel costs, stress and time 


Pinsonneault and 
Boisvert, 2001 


Positive impacts on individuals: 

• Reduction or elimination of commuting time (Baruch and Nicholson, 1997) 

• Potential to alleviate high levels of stress (McCune, 1998) 

• Increased flexibility is associated with higher levels of satisfaction and productivity. 


TABLE 2 

PROBLEMS WITH TELECOMMUTING 


Adam and Crossan, 
2001 


Organizational level: 

• Weakening corporate culture 

• Difficulties in remote supervising and management 

• Burdens in communication 

• Feelings of isolation 

• Work-family conflicts 

• Lack of technical and social support 

• Communication burdens 


Fitzgerald, 1994 


Employees who choose to telecommute are: 

• Not considered serious 

• Receiving less interesting and visible assignments 

• Receiving less feedback 

• Receiving little or no mentoring 
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Overall, the practitioner-oriented literature is contra- 
dictory in findings (McCloskey, 1998). Some studies 
state that telecommuting is a way to balance work- 
family conflicts and reduce stress, while other research 
reports that telecommuting leads to family conflicts, 
increased stress, and decreased productivity. Similarly, 
other studies report that organizations have not realized 
the much-anticipated cost reduction. In much of this 
literature these contradictions are a result of ambiguity 
in the definition of telecommuting, and in the research 
questions being investigated. 

To respond to these contradictions, researchers have 
highlighted the elements that should be described when 
studying and reporting telecommuting arrangements. 
McCloskey and Igbaria (1998), for example, state that 
the elements for studying telecommuting are: location, 
use of technology, employment relationship, and time. 
Location refers to the actual place from which an 
individual is telecommuting, such as his/her home, or a 
customer’s premises. Table 3 describes these 
components in further detail. Combining these different 
elements results in a different type of telecommuting, 
and thus in a different experience for telecommuters. 
For instance, the experience of a part-time employee 
who telecommutes to his office every day from a 
customer premise by means of computer access is 
different from the experience of a full-time employee 
who telecommutes two days a week from home. 
Consequently, these experiences cannot be compared 
without risking contradictions. 



TABLE 3 

COMPONENTS OF TELECOMMUTING 
DEFINITION 



Telecommuting 

Parameter 


Description 


Location 


Work done at a satellite location, 
branch office, customer premises, 
home. 


Use of 
Technology 


Technology use to maintain the 
link with the office, such as 
telephone only, computer only, 
fax, e-mail, or videoconferencing. 


Employment 

Relationship 


Type of contract between organi- 
zation and employee such as full- 
time, part-time, self-employed, etc. 


Time 


Amount of time that employees 
telecommute such as everyday, 
one day a week, two days a week, 
or other 



McCloskey and Igbaria urge researchers to specify the 
type of work arrangement that they are investigating to 
allow the comparison of research findings and the 
proper interpretation of results. Following this advice, 
recent academic researchers have narrowed down the 
definition of telecommuting and have often stated it as 
follows: telecommuting is working at home one or two 
days a week using computer and communication 
technology to maintain a link to the office (Belanger, 
1999; McCloskey, 2001). Thispresent studyadopts this 
definition. 

Academic researchers have also initiated efforts to 
empirically measure telecommuting outcomes and 
experiences while at the same time controlling for 
methodological issues that impair the comparison of 
results. One of these efforts is that of Belanger (1998) 
who contributed the following research framework for 
telecommuting (Figure 1). 

This model begins with four determinants for 
telecommuting success. Success can be analyzed by 
measuring the outcomes at three different levels. The 
determinants of success are: characteristics of the 
organization, characteristics of the individual, the type 
of work performed by the individual, and the type of 
technology required to do their work (see Figure 1 and 
Table 4). Belanger argues that these determinants must 
be matched to ensure succesful outcomes of 
telecommuting. For example, in another research 
project, findings supported the matching of computer 
and telecommunications technology to telecommuters, 
based on their needs to communicate with teammates 
(Belanger, Collins, and Cheney, 2001). 

Regarding the outcomes of telecommuting, Belanger 
states that telecommuting has impacts at three different 
levels: society, organization and the individual. At the 
individual level, she observes that telecommuting can 
provide “increased schedule flexibility, improved 
quality of work life, reduced commuting and clothing 
costs, increased job satisfaction, and reduced stress” 
(Belanger, 1999, p. 142). All these benefits combine to 
increase productivity, satisfaction, and performance of 
individuals. At the societal level, telecommuting is 
beneficial because it may help reduce pollution and 
traffic congestion, two important concerns for people 
living in large cities. Finally, at tiie organizatiaial level, 
telecommuting can reduce facilities and overhead costs, 
and increases productivity (Belanger, 1999). 
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FIGURE 1 

BALANCER’S FRAMEWORK FOR STUDYING DISTRIBUTED WORK ARRANGEMENTS 




Societal 




Organizational Individual 



TABLE 4 

VARIABLES THAT MAY IMPACT SUCCESS 
OF TELECOMMUTING 



Organizational Objective for establishing program 
** Culture 

** Control mechanisms 

** Objectives for participating 
** Skills 

** Communications requirements 
Coordination mechanisms 

Actual technologies 
** Physical environment (homeoffice) 
Security issues 

Variables marked with (**) will be included in the 
present research (Belanger, 1999). 

Belanger tested this research framework in two cases 
and has developed survey instruments to measure the 
impact of telecommuting mostly at the individual level. 
In one case, she compared the perceived productivity, 
performance, sense of personal control, and satisfaction 
between telecommuters and non telecommuters. She 
measured these outcomes using questionnaire items, and 
found evidence to sipport that telecommuting positively 
impacts productivity and personal control as perceived 
by telecommuters. She did not find evidence that 
telecommuting improves performance and satisfaction 
of telecommuters. 



Other studies have explored the outcomes of telecom- 
muting as well. Some have studied outcomes and 
individual experiences such as career advancement 
prospects, work-family conflicts, organizational commit- 
ment, and autonomy among others (McCloskey, 2001). 
McCloskey found that telecommuters experience more 
autonomy, less work family conflicts, and less career 
support than nontelecommuters, but that they do not 
experience any differences in terms of organizational 
commitment, satisfaction, and career advancement. 
Others have investigated additional outcomes such as 
turnover intentions, and have found that telecommuters 
are more likely to stay in their current jobs (McCloskey 
andigbaria, 1998). 

McCloskey’ s and Belanger’s studies are relevant to the 
present investigation. Both studies have measured the 
difference in outcomes between telecommuters and non 
telecommuters using the experiences of corporate 
employees working for a single organization. Moreover, 
both researchers acknowledged the limitations imposed 
by the samples they studied. Following the same 
research model and using a similar instrument, the 
present project investigates the difference in outcomes 
between telecommuters and non-telecommuters in 
samples from two different countries. The project uses 
a sample with different characteristics to expand the 
previous findings. For instance, McCloskey stated that 
the findings of her study might have been influenced by 
the fact that the organization from which the sample was 
drawn encouraged the practice of telecommuting among 



Individual 

Work 

Technology 
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its employees. Since Belanger’s respondents were 
mostly managers and professional employees, she 
recommended in further investigation the broadening of 
the sample to include other types of employees. 
Following these authors’ recommendations, this study 
targets corporate employees working for different 
organizations and performing different types of work 
within their organizations. The objectives are to 
investigate similarities in organizational support for 
teleworking in both countries and the effects of 
telecommuting in terms of productivity, satisfaction, and 
performance also in the two nations. 

RESEARCH QUESTIONS 

Overall, the literature indicates that the most significant 
benefit of telecommuting is increased productivity. 
Writers and researchers report findings on individual 
productivity gains from telecommuting in the range of 
30 to 100 percent (Adam and Crossan, 2001). Belanger 
(1999) also found evidence to support that telecom- 
muters “do more at home than at the office” (p. 149). 
Hypothesis 1 is important because of: 

• The importance of improving productivity and 
performance for organizations and individuals; 

• The amount of evidence reported in other studies of 
developed countries; and 

• The lack of research in telecommuting in Mexico. 

H 1 . Telecommuters in Mexico and the U.S. will report 
different telecommuting experiences regarding 
productivity, performance and satisfaction. 

Similarly, since organizational characteristics influence 
the organizational support or rejection of different tele- 
commuting alternatives, this project tests how much 
organizational support is found in Mexico and in the 
U.S. for telecommuting settings. The more organiza- 
tional support the higher the number of telecommuting 
experiences we can find. To study this alternative the 
project tests the following hypothesis: 

H2. Telecommuting in the U.S. receives more 
organizational support than telecommuting in 
Mexico. 



METHODOLOGY 

A questionnaire measuring the real and perceived 
outcomes of telecommuters was developed and tested 
using two samples. One comprised 1 1 1 employees of 
corporations in Southern California, and the other 
consisted of 66 employees in firms in Mexico City. The 
sample was drawn from senior and graduate students 
and from employees at four major universities. The 
participants were asked to provide answers to a total of 
49 items in a survey handed in class. Before 
administering the questionnaire in Mexico, it was 
translated into Spanish, tested in both languages, and 
adjusted and modified according to the feedback from 
three experts in the areas of telecommunications, 
information systems and management. It is worth 
mentioning that the authors are bilingual. 

The survey questions were grouped into six sections, 
four of them with answers on a 5-points-Likert scale that 
ranged from strongly disagree to strongly agree, while 
the other two sections asked participants to select their 
answers from a list of possible responses. 

The responses to the survey were recorded andanalyzed 
using SPSS version 8.0. The analysis of the data was 
conducted in three stages. First, the demographic 
characteristics of both samples were explored; second, 
the responses were divided into four groups (U.S. and 
Mexico and telecommuters and non telecommuters) and 
analyzed separately as independent groups. Under tiiis 
design, analyses of the telecommuters’ perceptions on 
productivity, performance, and satisfaction was carried 
out. Two-way cross-tabulations were applied to test 
associations of categorical variables. Pearson Chi 
Square was used to test for the significance of 
associations. 

ANALYSIS OF RESULTS 
Sample Characteristics 

In the case of the U.S. sample, the proportion of 
telecommuters in the total sample is 19.8 percent (22 out 
of 1 1 1) and for non-telecommuters is 80.2 percent (89 
outof 1 1 1). With respect to gender and age, the majority 
of individuals in the total sample were males (74 
percent) with an age-range between 1 8 and 25 years (67 
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percent). Chi-Square test shows no difference in the U.S. 
sample with respect to age and gender. In the case of 
Mexico, the proportion of telecommuters in the total 
sample is 45 percent (30 out of 66) and for non- 
telecommuters is 55 percent (36 out of 66). Similarly to 
the USA sample, the majority of individuals were males 
(50 out of 66). With respect to the age distribution, the 
Mexican sample is older, in particular the frequency of 
individuals in the range of 23 to 30 years old is much 
higher for Mexico versus the U.S. More relevant for the 
purpose of this project is the difference between 
telecommuters by country. Table 8 shows the that the 
two samples are significantly different in work 
arrangement. 

TABLE 8 

TELECOMMUTERS BY COUNTRY, 
INCLUDING UNEMPLOYED RESPONDENTS 



Telecommuters’ Productivity, Performance, and 
Satisfaction 

The second stage of the analysis of results presents the 
tests for differences in productivity, performance, and 
satisfaction between telecommuters in Mexico and 
telecommuters in the U.S. According to Tables 10, 11, 
and 12 telecommuters in the U.S. differ from those in 
Mexico in their self-ascribed productivity. Telecom- 
muters in the U.S. consider themselves as being more 
productive than those in Mexico. On the other hand, 
there are not other differences between telecommuters 
in Mexico and in the USA. 

TABLE 10 

CROSSTABULATION OF 
TELECOMMUTERS’ SELF-ASCRIBED 
PRODUCTIVITY BY COUNTRY 





No Tele 


Tele 


Total 




Disagree 


Agree 


Neutral 


Total 


USA 


89 


22 


111 


USA 


6 


58 


20 


84 


Mexico 


36 


30 


66 


Mexico 


14 


13 


7 


34 


Column Total 


125 


52 


177 


Column Total 


20 


71 


27 


118 



Chi-square = 13.109 df = 1 Prob=.000 Chi-square = 20.469 df=2 Prob=.000 



A more detailed analysis of the Mexican sample showed 
that 15 telecommuters out of 30 are unemployed, and 
they rather belong to a category of work at home than a 
telecommuting setting. This is a cultural difference, 
since during times of recession in Mexico, many middle 
class workers leave the formal labor force and join the 
informal labor force as self-employed workers. 
Sometimes, these self-employed workers form networks 
and alliances with each other. The survey was 
conducted in early 2002 at a time of economic recession 
in Mexico. Table 9 presents the result of a cross- 
tabulation, excluding unemployed individuals. The Chi- 
Square test shows that the difference between both 
telecommuting samples remains significant. 



Table 9 

Telecommuters by Country, Excluding 
Unemployed Respondents 





No Tele 


Tele 


Total 


USA 


89 


15 


104 


Mexico 


35 


14 


49 


Column Total 


124 


29 


153 


Chi-square = 4.340 


df= 1 Prob 


= 037 





TABLE 11 

CROSSTABULATION OF 
TELECOMMUTERS’ SELF-ASCRIBED 
PERFORMANCE BY COUNTRY 





Disagree 


Agree 


Neutral 


Total 


USA 


1 


13 


6 


84 


Mexico 


4 


18 


8 


34 


Column Total 


5 


31 


14 


51 



Chi-square = 2.378 df=3 Prob=.498 



TABLE 12 

CROSSTABULATION OF 
TELECOMMUTERS’ SELF-ASCRIBED 
SATISFACTION BY COUNTRY 





Disagree 


Agree 


Neutral 


Total 


USA 




16 


5 


21 


Mexico 


4 


21 


5 


30 


Column Total 


4 


37 


10 


51 



Chi-square = 3.187 df = 2 Prob = .203 
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Organizational Support in Mexico and the U.S. 

Another important factor in the telecommuting decision 
is whether or not the organization supports different 
working arrangements. To test this research question on 
organizational support, the survey asked respondents to 
indicate the level of management support to telecom- 
muting and whether or not there were other employees 
working under teleworking settings. Tables 13 and 14 
show that there is no difference in the extent of 
organizational support for telecommuting between 
companies in Mexico and the U.S. 

TABLE 13 

CROSSTABULATION OF PRESENCE OF 
ORGANIZATIONAL SUPPORT FOR 
TELECOMMUTING BY COUNTRY 







Disagree 


Agree 


Total 


USA 


Count 


31 


31 


62 




Exp. Count 


28.7 


33.3 


62.2 


Mexico 


Count 


20 


28 


48 




Exp. Count 


22.3 


25.7 


48.0 


Total 


Count 


51 


59 


no 




Exp. Count 


51.0 


59.0 


110.0 


Chi-square 


= .756 df= 


1 Prob = . 


385 





TABLE 14 

CROSSTABULATION OF PRESENCE OF 
ORGANIZATIONAL PRACTICES FOR 
TELECOMMUTING BY COUNTRY 







Disagree 


Agree 


Total 


USA 


Count 


35 


47 


82 




Exp. Count 


37.0 


45.0 


82.0 


Mexico 


Count 


25 


26 


51 




Exp. Count 


23.0 


28.0 


51.0 


Total 


Count 


60 


73 


133 




Exp. Count 


60.0 


73.0 


133.0 



Chi-square = .519 df = 1 Prob = .475 



Table 15 summarizes the research findings. According 
to this table telecommuters in the U.S. and Mexico show 
no difference in telecommuting performance, telecom- 
muting satisfaction, and to the extent to which 
organizations facilitate teleworking arranganents. Only 
hypothesis 1, that telecommuting productivity varies by 
country, is supported. 



TABLE 15 

SUMMARY OF FINDINGS 



Outcome 


Chi- 

Square 


Sig 

(2-tailed) 


Supported 


Performance 


2.378 


.498 




Productivity 


13.10 


.000 




Satisfaction 


3.187 


.203 




Org. Supp 


.765 


.385 




Org. Prac. 


.519 


.475 





DISCUSSION 

This investigation explored the differences between tele- 
commuters in the U.S. and Mexico in terms of 
productivity, performance, satisfaction, and presence of 
organizational support. Telecommuters in Mexico and 
the U.S. report similar perceptions in their telecom- 
muting performance and satisfaction, but not for 
productivity. There were not country differences in 
organizational support and the practice of telecom- 
muting. Telecommuters in the U.S. consider that they 
are more productive due to their working arrangement. 
Mexican telecommuters disagree that telecommuting 
enhances their productivity. This result is even more 
contradictory if we consider the proportion of telecom- 
muters in the Mexican sample. 

Differences in economic development, information 
technology infrastructure, availability of computers, and 
education suggests a higher proportion of telecommuters 
in the U.S. sample than in Mexico. However, the 
number of telecommuters in the Mexican sample is 
higher, proportionally, than the number of tele- 
commuters in the U.S. one, and yet the Mexican 
telecommuters do not consider themselves more 
productive. 

Lack of difference in the organizational support for 
telecommuting between the two nations is another 
important finding. From the Mexican perspective, 
Mexican legislation and Mexican culture promote a tight 
relationship between employees and managers. The 
apparent support for flexibility in this relationship was 
unexpected. There are two potential explanations: (1) 
cultural change due to economic liberalization policies 
and (2) high unemployment has generated new labor 
relationships in Mexican organizations. Kramer and 
Dedrick (2000) suggest that changes in economic 
policies have promoted the development of the IT 
industry in countries like Mexico, Brazil, Turkey, and 
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India. In the case of Mexico, economic liberalization 
programs started in the late 80s. A decade later, there 
was more IT infrastructure, computers, telephone lines, 
and IT specialists. This factor may explain the new 
attitude of organizations toward telecommuting. The 
other trigger of new organizational structure may be the 
high rate of unemployment at the time of the survey. 
This unemployment rate has likely triggered new 
organizational structures. 

LIMITATIONS AND RECOMEND ATIONS 

Previous researchers have acknowledged limited 
generalizability of their findings due to the fact that 
samples were drawn from a single organization or 
because sampled individuals performed the same type of 
work. This investigation tried to address these 
limitations by surveying individuals who work for 
different organizations and who perform different types 
of job. However, this investigation is not exempt from 
limitations. 

One limitation is the restricted demographic profile of 
the samples. Many of the individuals in the sample 
were individuals with age-range of 18 to 30 yeais, and 
had limited working experience. The International 
Telework Association and Council’ annual survey 
reported that 

the typical teleworker ... is a college-educated white 
male between the ages of 35 and 44, married and 
earning at least $40,000 per year. . . . 

Therefore, also targeting this population would provide 
broader information on the outcomes of telecommuting. 
Findings in the Mexican sample point toward this 
direction. A sample with major average age prompted 
a bigger telecommuters population. 

Another limitation was imposed by definition of 
telecommuting utilized. This research framed telecom- 
muters as individuals who work from home one or two 
days a week using computers and communications 
technology to maintain links to their offices. Some 
individuals in the sample reported being left out of the 
telecommuter category because, although they work 
from home one or two day a week, they did not use 
computers at home. Without a doubt the proportion of 
telecommuters would have been greater if these 
individuals were included as telecommuters. 



Lastly, a major drawback to this project is that the 
Mexican sample may be non-representative of the 
telecommuting phenomena in Mexico. The rationale 
behind the sampling procedure was to avoid sampling 
limitations for telecommuting in Mexico, due to lack of 
education, of computer skills, and of access to 
computers. Sampling senior undergraduate and graduate 
students succeeded in neutralizing those factors. 
However, the Mexican sample characteristics are not 
representative of the entire national population of 
Mexico. 

CONCLUSIONS 

The results of this investigation concern the differences 
in outcomes resulting from alternative working 
arrangements. Telecommuters were found to perceive 
themselves as having higher levels of productivity than 
for the U.S. sample. No evidence of difference between 
the Mexican and the U.S. samples regarding satisfaction 
and performance were found. Also the results found no 
difference between the U.S. and Mexican organizational 
support for telecommuting arrangements. 

Additionally, this research used the framework 
developed by Belanger for the study of telecommuting 
in samples belonging to two countries. This project’s 
findings encourage future research comparing telecom- 
muting arrangements in Mexico and the U.S. 

Telecommuting has been shown to be a challenging 
research issue due to the many factors involved in its 
practice (i.e., organizational characteristics, individual 
characteristics) and the many levels this arrangement 
may impact (i.e., society, organization, individual), as 
well as to the difficulty in targeting the right population. 
Additional delimitation of the individual characteristics 
of telecommuters needs to be emphasized to make them 
the target of further investigation. Much has been said 
about the rise and demands for knowledge wcrkers and 
the benefits of expanding corporate boundaries to reach 
further workforce and skills. Telecommuting and the use 
of information technology to eliminate physical com- 
mutes have been demonstrated as viable options to 
increase productivity and corporate reach. Identification 
of the circumstancess that have led Mexican telecom- 
muters to ascribe themselves as less productive should 
be investigated further. 
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PROPORTION, PEDAGOGY AND PROCESSES 
THE THREE P’S OF E-LEARNING 
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ABSTRACT 

There is a tendency to equate electronic learning or e-learning with distance learning. In fact, e-learning covers a 
broad spectrum, from learning which is primarily contact based to learning which is 100% distance. Thus, each 
course can be measured by the proportion of learning and teaching that is intended to be conducted electronically. 
The principles of course design applied to the development of a given course should be influenced by the position of 
a course on this spectrum. Furthermore, there is a relationship between the proportion of e, the design strategy and 
the pedagogic model adopted by the designer. In this context we juxtapose didactic and constructivist models of 
learning. The substantial recent development of web based learning has sparked renewed interest in constructivism 
and the way in which web based technology can facilitate engagement. Whilst the relationship is not linear, we argue 
that the greater the proportion of e-learning used, the more developed the active learning components that are 
required. Finally, the development lifecycle adopted, and therefore the processes used in the development of the 
course and the software used for implementation of the course, will differ according to the proportion of e-learning 
anticipated. This argument is developed in the context of e-learning in higher education. 



INTRODUCTION 

We can define e-leaming simply as learning facilitated 
through electronic means. Research and development in 
e-leaming in higher education has been well established 
for decades although the term e-leaming is relatively 
new. In the 1980s and 1990s terms such as Computer 
Based Learning (CBL) and Courseware were widely 
used, yet fit the definition above. There has been a 
substantial increase in interest in higher education in e- 
leaming since the establishment of the World Wide Web 
(Web) in 1991, and especially since the late 1990s as a 
result of the widespread adoption of the Web as a 
universal interface. Thus a contemporary definition of e- 
leaming should probably include the use of the medium 
of the Web since this is how the term is now used 
colloquially. 

At the same time we cannot ignore the use of other 
terms used in this context. The well established term 
online learning has to some extent been superseded by 
terms including virtual learning, web based learning, 
open learning, flexible learning, mixed mode learning 



and blended learning. Although these terms are well 
established their definitions are not. We use e-leaming 
simply because it is probably the most widely used of a 
number of terms with similar meanings. 

Today we are in the midst of a stampede to embrace e- 
leaming with significant investment being made by 
Universities and other organisations throughout the 
world. Major alliances have been established, such as 
Universitas 21. Universities are both keen to defend and 
extend their empires. There has been a msh to place 
learning materials on the World Wide Web for fear of 
‘missing the boat.’ Most Universities have appointed 
staff with titles including the terms used above. Since 
this initial msh, there has been a substantial body of 
opinion questioning the educational benefit of this 
development, subjecting e-leaming to a level of scmtiny 
not experienced by more traditional means of 
dissemination of knowledge. Pedagogical quality has 
become the litmus test of e-leaming and the quantity and 
calibre of interaction has become one of the main 
measures of this quality. 
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WHAT PROPORTION OF E 

It is valuable for the course designer or lecturer to decide 
upon the proportion of both the delivery of their course 
and the student learning experience that will be 
conducted by electronic means. This decision depends 
upon a great number of often conflicting issues including 
course objectives, stakeholder views and resources. 

The great majority of higher education includes some 
use of electronic materials. This can range from the use 
of copies of electronically produced lecture notes, to the 
use of slides projected through computer equipment to 
entire courses on the World Wide Web. A simple 
horizontal rule can be used to indicate the proportion of 
the delivery of a course that is intended to be conducted 
electronically (Figure 1). Note that this line may be 
applied to either a measure of the proportion of 
electronic delivery or the proportion of the student 
learning experience: no distinction is drawn in this 
paper. By definition a course which includes e-leaming 
may also fall anywhere on the line in Figure 1. 

We can attempt to add to this scale different terms, 
whose use to some extent can be distinguished by their 
proportion of electronic delivery. We can call this a 
spectrum of e-leaming. 

Unlike the spectrum of light the scale in Figure 2 has no 
rigid scientific basis since terms such as flexible 
learning, open learning, online learning and distance 
learning tend to be used interchangeably and receive 
meaning only through their useage. For instance a course 
whose content is in the main delivered electronically 
may well be described as flexible learning. It is not the 
intention of this paper to provide a classification of these 
terms. 

Whilst it is important for course developers to determine 
where their course falls on this scale, clearly in practice 



many courses evolve along the scale from left to right or 
right to left. Courses which are described as ‘full time 
attendance’ or contact teaching may include a substantial 
and growing element of distance learning by electronic 
means. Courses that are described as ‘distance learning’ 
may include the introduction of substantial face to face 
contact such as regional tutorials. The process of courses 
moving along the scale is gradually blurring the 
distinction between contact and distance learning. 

SHOVELWARE 

The use of electronic media is not new: the development 
of the World Wide Web (Web) has simply extended the 
range and fimctionality of facilities used. The Web has, 
however, made the development of e-leaming much 
more widely available. 

There has been a tendency amongst early adopters of 
Web based learning to use the Web as a simple huge 
repository for course materials such as lecture notes. 
Once such materials were placed on the Web it could be 
called e-leaming and since the term was synonymous 
with distance learning, traditional means of teaching 
such as lectures and tutorials ceased to be required. 
Accountants mbbed their hands in glee until questions 
were raised concerning the pedagogical value of such 
education. This practice has became known as 
shovelware. Shovelware refers to "any content shoveled 
from one communication medium to another with little 
regard for the appearance, ease of use, or capabilities of 
the second medium" (Fraser 1999). There has rightly 
been a strong reaction to shovelware. Fraser argues "You 
may have shifted the nature of student access by moving 
to the Web, but access is not insight". There is 
substantial evidence that the students don’t want it (Kara 
and Kling 2000), (Oliver and Omari 2001) and (Sheard 
et al 2000). There is also significant concern about the 
consequent commercialisation of higher education 
(Noble 1998). 



FIGURE 1 

PROPORTION OF DELIVERY BY ELECTRONIC MEANS 



► 



0% Electronic 



100% Electronic 



196 



Proceedings of the IT^ Annual Conference of the International Academy for Information Management 



190 



FIGURE 2 

SPECTRUM OF E-LEARNING 
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In her paper ‘Models of Online Courses’ (1998) Robin 
Mason discusses how experience of using Web 
technology has shown course designers how learning is 
best encouraged and supported in the online 
environment. She uses the term ‘Pedagogical 
Evolution’ in this context. In consequence to this 
reaction there has been a scramble to adopt ideas 
concerning good pedagogical practice from the obvious 
source of knowledge; experts in distance education and 
those with experience in the development of CBL. In 
Britain as in other countries this has meant recourse to 
our specialist distance learning provider, The Open 
University. The fruits of this experience are now being 
applied to e-leaming and may typically include the 
following ideas: 

1. Since effective learning involves active learning, the 
pedagogical quality of e-leaming materials can be 
simply measured by the quality of interaction 
involved. 

2. Development of effective e-leaming materials is 
extremely expensive and is best left to experts. One 
Figure oft quoted whose source is unclear is that 200 
hours of development time are necessary for one 
hour of delivered e-leaming material. The Open 
University once estimated that the development of 
a new course cost £1 million (Scott 2001). 

3. As a result of 1 & 2 above, substantial planning, 
analysis and design is necessary before e-leaming 
materials can be developed. By implication 
therefore a sequential lifecycle model is the most 
appropriate to the development of e-leaming. (eg see 
Phil Race model used in ‘course design for On- 
line learning’ 1 day workshop at University of 
Salford 3/11/00 Figure 7. Also see Dick and 
Carey(1990)). 

4. Off the shelf tools do not provide the level of 
complexity or customisability required and are 



therefore not suitable for the development of e- 
leaming. 

This pedagogical correctness is examined in the 
following sections. 

THE PROPORTION OF INTERACTION 

It is widely agreed that the level of engagement with and 
participation by students is a key metric in the quality 
assessment of higher education eg (QAA 2000 p. 41) 
and (SCOTT 2000). Designing interaction is rightly 
regarded as a vital part of the experience of higher 
education. It follows therefore that the greater the 
proportion of the learning objectives that are expected to 
be achieved by electronic means the greater the 
proportion of interaction that has to be designed into the 
e-leaming materials. It can be argued that the converse 
is also true (see Figure 3). 

ELECTRONIC LEARNING BY SPECIALISTS 

It has been argued above that e-learning is not 
necessarily the same as distance learning. It follows that 
in some situations, where the delivery of courses whose 
delivery is in great part (eg >90%) electronic, the 
development work may best be left to specialist distance 
institutions. In the majority of situations the bulk of e- 
leaming is lower on the spectrum and the main 
pedagogical issues may not be those involved in 
distance learning. 

Distance learning clearly offers substantial advantages to 
the learner, for example in the area of accessibility, and 
therefore has a substantial market. One could argue that 
courses whose learning and teaching which is mainly 
required to be conducted by distance should be left to the 
specialist distance learning institutions. They have the 
necessary infrastmcture such as departments for course 
design, reprographics, copyright, telephone helpline, on 
line libraries etc. The Web offers new 
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FIGURE 3 
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possibilities in terms of the media used in the delivery of 
such services. 

However, in general what the materials of the dedicated 
distance learning institution gain in quality of 
production, they lose in flexibility. When a course 
involves investment of 200 hours of labour per hour of 
delivered material, substantial and restrictive version 
control procedures are necessary. Furthermore, changing 
distance learning materials is a significant problem 
when students may be enrolled on a course for many 
years. In fast changing academic areas, such as 
Information Systems, this creates a substantial and 
intrinsic problem for distance learning. 

In this way institutions whose foundations are built upon 
contact teaching may usefully adopt an evolutionary life 
cycle approach for the development of e-leaming 
materials. 

LIFECYCLE MODEL TO BE USED 

The debate concerning development lifecycles, 
established in the literature of software engineering eg 
(McConnell 1 996), has been adopted into more general 
Project Management texts such as the Project 
Management Body of Knowledge (Project Management 
Institute 2000). There is significant debate concerning 
this issue in the context of web development - see 
(Scharl 2000) and (MacCormack 2001). One can 
juxtapose the sequential model of development known as 
the Waterfall model (Royce 1970) with the iterative 
model of development known as the Spiral model 
(Boehm 1988) and 1996). The evolutionary model put 
forward by (Scharl 2000) could be described as a close 



cousin of the spiral model. In the Waterfall model each 
stage of development follows each other sequentially 
until the final product is tested and released to the 
market. In the Spiral model, there is a substantial degree 
of user involvement and the product passes through a 
number of iterations before final release. In true 
evolutionary development there may be a number of 
planned releases, each one building upon the previous 
release. This is practised by a number of e-commerce 
organisations such as Expedia (ref Computer). There 
has been much debate on the respective merits of 
different life cycle models (McConnell 1996), (Redmill 
1997). In truth each is suited to different situations. 
Whilst the sequential model is the only practicable one 
for a major one off project such as a space mission to 
Mars which must be right first time, an Evolutionary or 
Spiral model may be much better suited to the 
development of a web site used for promotional purposes 
or for the purposes of supporting materials for a course 
that is primarily contact based. 

In this context one could argue that the life cycle model 
to be adopted depends upon the proportion of the 
learning objectives that are to be achieved by distance. 
Whilst the sequential model is most suitable for a course 
that is designed to be delivered mainly by distance, the 
spiral model or evolutionary model is better suited to a 
course that is primarily contact based and is using 
electronic materials for support purposes. This is of 
course an important issue since the emphasis in the 
Waterfall model on getting the analysis and design right 
at first may actually obstruct the introduction of valuable 
support materials, where the adoption of a ‘Just do it’ 
philosophy may be more appropriate see Figure 4. 
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ABSTRACT 

Introduction to Information Systems is a required second year course for all students completing an undergraduate 
degree in business or a business minor at the Sprott School of Business, Carleton University. This paper highlights 
recent curriculum innovations that have occurred in the course to incorporate concepts pertaining to enterprise 
systems, enterprise integration, process, and the transformation to e-business. The concepts are introduced in 
lecture and are then brought to life through collaborative learning activities as the students become a company 
(240co) and work in groups to simulate the concepts they have learned. At a second year level, students have not 
yet encountered all of the functional areas of a business nor have they seen the interplay among the functions. 
Through the simulated company experience, students come to appreciate what all the business functions are, and 
they experience some of the challenges a firm goes through on the road to becoming an integrated, e-business 
enterprise. 



INTRODUCTION 

Today concepts like e-commerce, e-business, intranets, 
and enterprise integration which only a few short years 
ago were scarcely found in IS curriculum, are working 
their way into the basic introductory information system 
textbooks reflecting what has been happening with IS 
faculty as they revamp and revise their classes to 
incorporate the new concepts. These concepts are not 
about IT along but include the need to understand 
business processes and the interaction among business 
functions. Consider for example learning how 
Marketing, Accounting, Production, and Operations 
interact and how they use technology in order to fulfill 
a customer’s order. 

The paradigm of enterprise integration and e-business 
suggests a move away from functional, silo based 
approaches when teaching these concepts (Selen, 2001 ; 



Shtub, 2001.) The reality is however, that the silos 
themselves are not disappearing even in organizations 
that claim to be process-oriented, fully integrated, and e- 
enabled. The implication for the students is that they 
must then be able to weave together the functional 
orientation of the firm with an overlay of integration 
through process, information flows, and information 
technology. This is consistent with research on skills for 
IS professionals which found that business knowledge, 
including an appreciation of business functions, and the 
ability to interpret business problems and develop 
appropriate technical solutions represent required 
critical skills (Noll and Wilkins, 2002). 

One of the modes of bringing the concepts of enterprise 
integration and e-business to the classroom has been 
through the introduction of enterprise systems also 
referred to as enterprise resource planning systems 
(ERP). Through programs like the SAP University 
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Alliance, universities world-wide are creating an 
environment in which students can use an e-enabled 
enterprise system like SAP R/3 to learn some of the 
concepts. A number of papers have been written that 
give examples of innovative ways of using enterprise 
systems in the classroom. Shtub (2001) for example 
presents a methodology for teaching processes along 
with a special training aid called the Operations Trainer 
that “simulates the entire order fulfillment process from 
customer orders to the purchasing of raw material. Four 
interconnected functional areas are presented. ” (p. 569). 
The focus is on the traditional management of 
operations in a functional organization with process and 
advanced information systems being emphasized. 
Another example is an industry-oriented initiative 
developed in Australia (Stewart andRosemann, 2001). 
Working with an industry partner, business and 
information technology graduate students develop an 
ERP-related reference process model of an actual 
business. “This project allows students to understand 
common business functions and appreciate a process 
view of organizations” (p.240). 

The Sprott School of Business at Carleton University is 
also embarking on introducing enterprise integration and 
e-business concepts into the classroom. The School 
uses a broad based definition of e-business — one that 
emphasizes that e-business is as much about business 
process and management practice as it is about 
technology. The definition includes the notion of an 
integrated enterprise, linked with its customers and 
suppliers, and with the ability to manage processes and 
information flows through the use of enterprise systems 
and Internet-based technology. 

Important in the School’s approach to e-business is 
recognition of the need to think cross-functionally. 
Integration cannot take place if the perspective remains 
one of functional silos. This is not to say that the 
standard functions of business such as Marketing, HR, 
Production, etc. will dis^pear. Rather, from a 
curriculum standpoint, the goal is to have the students 
appreciate both the functional perspective and the 
interdependence and interaction which takes place 
across functions in order to ccmplete the core business 
processes of an organization. 

At the undergraduate level, this goal has been 
particularly challenging given the limited knowledge the 
students have of the various business functions. By the 
time a student graduates they are expected to have some 
depth of knowledge in a particular function such as 



Accounting or Information Systems and it is expected 
that they will have touched all of the business functions 
through their required courses. Students in first and 
second year however, have had an introduction to only 
a few of the traditional business functions. It is these 
same students that are being introduced to the concepts 
of enterprise integration and e-business and that 
represent the audience for the curriculum innovation that 
is the focus of this paper. 

COURSE BACKGROUND 

Introduction to Information Systems is a required 
second year course for all students completing an 
undergraduate degree in business or a business miner. 
The course runs for one semester of 13 weeks. 
Approximately 600 students a year take the course in 
sections varying in size from 60-70 students each. This 
undergraduate business program core course serves as a 
strong building block for upper-level courses in all 
functional areas of the business school. 

As an introductory course, the range oftopics covered is 
very broad. The goal is not dq?th in a particular IS 
topic, but rather breadth across a range of IS related 
topics that reflect the use of information systems 
technology in business. Topics include but are not 
limited to: enterprise and functional systems; networks 
and telecommunications; databases and data enhanced 
decision-making; e-commerce and Internet 
technologies; business processes; and systems 
development and management. Hands on components 
include the development of simple, individual websites 
and as well as collaborative, group-based websites and 
an Access database that forms the backend for role- 
based access to data through an intranet. 

CURRICULUM INNOVATION - 240CO 

While the concepts such as enterprise systems, process, 
and e-business have been covered in this intro course for 
a number of years, there was a desire to create some 
mechanism to bring these concepts to life for the 
students. Telling the students that it is difficult to share 
data across the organization carries little significance. 
The goal of the curriculum innovation was to bring a 
business and all its complexity to the classroom so that, 
through a simulation, the students could at least in part 
experience some of the dynamics of a business as it tried 
to transform itself into an integrated e-business 
enterprise. 
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In the first class meeting, students are introduced to 
240co and are welcomed as employees in the company. 
All sections of the course run as their own company, 
though the basic company is the same for each section. 
The organization diart is shared with the students along 
with background on the history of the company and the 
products it manufactures. At this beginning stage, the 
firm does not do e-commerce, nor do they have an 
enterprise system. The instructor (Executive Suite) is 
the most senior person in the organization. Below the 
instructor are two levels of hierarchy — a senior 
management group and a series of functional area 
groups. In week two, the students are randomly 
assigned to one of the groups. In total 14 groups are 
created representing the following: Senior Manage- 
ment, Sales, Marketing, Customer Service, HR, 
Training, Finance, Accounting, Research and 
Development, Manufacturing, Engineering, Purchasing, 
Information Systems, and Shipping, Receiving and 
Warehousing. The potential for overlap in the definition 
of these functional areas is purposeful as one of the first 
exercises the students do when they get into their 
functional area group is to define the roles and 
responsibilities of the function. As students report out 
on how they have defined their function, overlaps in 
responsibilities are highlighted and the students are 
asked to reflect on what types of problems and benefits 
they think that might mean for the company. 

Table 1 outlines the series of activities that are covered 
over the course of the semester. The order of the list of 
activities reflects the order in which they are covered 
during the semester. Each activity is described briefly 
along with the outputs expected from that activity. The 
activity may take place directly and entirely within the 
time allotted in a weekly class or it may require the 
students to follow-up outside of class. Each section 
meets once a week for 3 hours. In most weekly sessions 
the first half of the class is spent covering the concepts 
in the textbook for that particular week. The second half 
of the session involves what is fondly referred to as a 
“group hug”. During the group hug, students work in 
their groups on the activity at hand. The activity may 
require them to interface with other groups. It may 
require them to interface with the senior management 



group. The groups may or may not do a brief report out 
in that same week of what they have worked on in their 
group. 

The most intricate of the simulation activities is the 
database exercise. The exercise is designed to highlight 
the difficulties of sharing data between functional areas 
in a large organization. In this assignment, each 
functional area group is required to request a meaningful 
set of data from three other arbitrarily designated 
‘partner’ groups. The requesting group asks a data 
question of their partner group (the Sales group, for 
example, might ask the Shipping, Receiving and 
Warehousing (SRW) group to report on current 
warehouse inventory levels of various products). The 
recipient group then creates a fictional set of data to 
answer the question. The recipient group must also 
write a set of queries in Access in order to answer the 
question in a meaningful way. These queries are then 
added to the Access database in a query table. The final 
step is that the recipient group needs to specify viewing 
permissions in another table. So if Sales requested data 
from SRW, then Sales, in creating the table, must give 
access to themselves, to the SRW group, to the Senior 
Management Group and to the Executive Suite. 

Once the databases are received, the instructor imports 
all the data from the individual databases using a set of 
routines that rely on the specific format of the student 
databases. The result is a master database that serves as 
the backend to an intranet site, and allows role-based 
access to the data that each group has been allowed to 
see. So an individual student logs into the 240co 
‘intranet’ website using their student number. The 
website has embedded logic that allows it to determine 
which area group a student is a member of, and then 
goes about creating an environment for the student, 
driven by their group membership. In the specific 
student’s environment they see only the data they are 
allowed to see, based on the permissions that were set as 
part of the database assignment. So the Sales Group 
would only be able to see the data that has been 
explicitly exposed to them through the permission table 
created as part of the assignment. 
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TABLE 1 

THE SERIES OF ACTIVITIES USED TO SIMULATE A FIRM 
TRANSFORMING ITSELF TO AN INTEGRATED E-BUSINESS ENTERPRISE 



240co 
Simulation 
Activities C/A* 


Description/Objective 


Outputs 


Evaluation 

Y/N 


Creation of 

Organization 

c 


Provide context oforganization. 


Put students into functional area groups 
including asenior management group. 


N 


Function’s Work 


Each functional area group is asked to 


Students get started in class and then in 


No - but 


System Definition 


define the roles and responsibilities for 


the next week are asked to make a brief 


written 


C+A 


their respective groups. The definitions 
are to include a description of the basic 
data that the group would create, who 
their customers are ,what technology they 
would use, and wh at their main functional 
processes are. 


(5 minute) presentation on the 
definitions they have derived for their 
function . 


feedback 

provided 


Employee Web Sites 


Creating a website that reflects student 


2 -3 page website created by each 


Yes- 


A 


and student’s role as an employee in 
240Co. 


student. 


Individual 

assessment 


New C orporate 

Strategy 

C 


The Executive Suite (instructor) lets the 
students know that the company is going 
“e”. The firm is now going to develop e- 
commerce capabilities, redesig n 
processes, and installan enterprise system. 


Brief(2-3 minute) in-class presentation 
from each functional area group of what 
impact they perceive the recent 
announcement will have on their 
function. 


No 


Identifying and 


Each functional area group identifies the 


Brief (2-3 minute) presentation from 


No — but 


Sharing Business 


business data they would own. They also 


each functional area group of which 


debrief w ith 


Data 

C 


identify which functional area groups 
would most likely request data from them 
and which functional area groups they 
would most likely request data from. In 
all instances examples of the data are to be 
supplied. 


other function al group s they wo uld 
likely go to for data to run their 
operationseffectively and which groups 
would likely come to them for data. All 
functional groups are listed on the 
board. As the groups reportout, arrows 
are drawn between the groups to 
representthesharing of data. The result 
is a messy spaghetti chart. 


whole class. 


Functional Area 
Group Web Sites 
A+C 


Focus on creating an intranet and 
contrasting desire to be unique at 
individual and functional area group 
levels and yet look for consistency at a 
corporate level. 


Functional Area Group Web sites linked 
into an intrant. Websites to include a 
mission statement of function along 
with main roles and products or services 
of function. Presentation of websites 
by respective functional groups. 


Yes 


Data Exchange / 
Intranet 
Development 
A+C 


Using Access students create data tables 
and queries on data that they create that 
relates to their function. 


Students create a set of internal tables 
(for their function) as well as external 
tables thatarebasedon requests fordata 
from 3 other functional area groups. 


Yes 


Future Directions 
A+C 


Each function area group is asked to 
reflect back on the semester and identify 
which of the technologies that have been 
discussed in class would help to move the 
e-agenda forward for their specific 
function and the company as a whole. 


Five minute prepared presentation from 
each functional area group. 


Yes 



* Context of Activity C = Classroom, A = Assignment 
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BEST COPY AVAILABLE 



CHALLENGES AND LIMITATIONS 

The main challenge of making 240Co possible each 
semester is the administrative requirement of running 
the course. The administration includes both the face- 
to-face activities with the students along with the 
backend systems functionality associated with the 
course website. On the student front, managing the 
groups and processing the assignments can require 
considerable instructor attention. Well-trained teaching 
assistants to run the labs for the hands-on components 
are also a must. The teaching assistants have 
contributed considerably to the creation of documents, 
available through the course website, that serve as aids 
for each of the assignments. 

On the technology front, the success of the Access 
assignment relies heavily on a custom-built website 
using Active Server Pages to embed processing logic in 
the website. The student interface, which is evolving as 
the course matures, allows for news items, assignment 
information, course information, lab materials and other 
associated documents to be disseminated in a one-stop 
shopping format. In addition, student marks and group 
information is hidden behind a usemame4>assword 
protected front-end that also serves to provide the logic 
to the intranet site for the Access assignment. This 
importance of the website backend cannot be 
underestimated. Not only is it an efficient vehicle to 
provide course-specific materials and secure information 
to students, it also serves to illustrate many of the 
concepts taught in the course. 

The course is currently designed for sections of 
approximately 70 students. This may not be large 
enough to satisfy class sizes in other institutions. The 
simulation could be expanded to accommodate greater 
numbers of students through the addition of more 
functional areas groups. The current number of area 
groups is 14. A maximum of 16 area groups is 
suggested with no more than 6 students per group 
bringing the possible section size to 96. Beyond this 
size, the quick report outs and coordination across 
groups would become unruly. 

FUTURE DIRECTIONS 

The simulation of 240co serves to demonstrate many of 
the concepts behind enterprise systems including 
business processes, shared data stores, etc. A hands-on 
component using an enterprise system is a logical next 
step. Beginning in the fall of 2002, the use of SAP’s 



R/3 enterprise system will be incorporated into the 
course curriculum. Each functional area group will 
enter transactions into the system that are logical for 
their function. The students will then be required to 
track which functional area groups have been impacted 
by the transactions. The eventual use of mysap.com will 
provide new opportunities to highli^t additional e- 
business concepts. The goal is that the 240co simulation 
will bring each of the organizational functions to life for 
the students and the interfacing with the enterprise 
system will serve to reinforce the interaction and 
interdependence among the functions in order to 
accomplish core business processes. 

Assessment of learning outcomes is anecdotal at this 
point. Attendance has been a problem in this course in 
the past. The new format has resulted in near full 
attendance at all sessions. The level of activity in the 
classroom suggests that students are grasping the 
complex concepts. They get down to business in their 
group hugs and are very serious about the discussion. 
As part of their functional area group websites the 
students must create a mission statement and a 
definition of the roles of their functioa They take this 
exercise very much to heart and are quite proud to share 
with their classmates the contribution they believe their 
function is making to the success of 240co. The 
students often take initiative within their functional role 
to suggest some new options for 240co. These 
suggestions are beyond any of the requirements of the 
assignments and demonstrate the ownership that the 
students feel toward 240co. More formal assessment of 
this curriculum innovation is required and will be a 
focus in the coming semester. 

CONCLUSIONS 

The management concepts behind e-business and 
enterprise systems are as important as the technologies 
associated with the concepts. Through the use of the 
240co simulation, second year students get to know all 
the functions better and they begin to see how the silos 
operate and how they need to interrelate with each other. 
At the same time they also experience how technology 
can help and hinder the flow of information necessary in 
an integrated environment. 

The simulation can be used on its own or as an 
intermediate step to bringing R73 into the classroom. 
Installing SAP’s R73 enterprise system into a university 
setting is not a trivial exercise. As a consequence, an 
additional goal of this course innovation was to develop 
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a way to have students leam about enterprise integration 
and e-business without dependence on having an 
established R73 environment. 
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Information systems (IS) education can be taught across 
many types of schools and departments including 
Management, Engineering, and Liberal Arts. Yet, 
regardless of type, IS curriculum models play a key role 
in the revision and modernization of university curricula 
and textbooks to support them. Many factors influence 
what students are ultimately taught, including 
curriculum models, accreditation requirements, student- 
body characteristics, instructor, instructional materials, 
and requirements of the business community. It is 
generally agreed that IS curricula need frequent 
updating to remain effective. 

IS curriculum models not only enable academic 
communities to maintain programs that are consistent 
both with the needs of the business community and the 
body of IS knowledge, but also provide for 



administrative awareness regarding course offerings and 
resources needed for a viable program, including 
computing hardware, software, and laboratory. Those 
who hire students graduating from programs following 
a curriculum model have a better understanding and 
appreciation for each student’s knowledge base. 

There are several published curriculum models for 
undergraduate IS education in a four year degree 
program. The IS ’97 Model Curriculum and its ongoing 
revision, IS’2002 Model Curriculum [http://www 
.is2000.org/], are independent of academic unit and 
corresponding department. The IS’97 Model Curri- 
culum, developed jointly by the Association for 
Computing Machinery (ACM), the Association for 
Information Systems (AIS), and the Association of 
Information Technology Professionals (AITP) (formerly 
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DPMA), was reviewed by over a thousand individuals 
from industry and academia. The IRMA Curriculum 
Model was developed by a joint task force of the 
Information Resource Management Association (IRMA) 
and the Data Administration Management Association 
(DAMA) and was adopted at the 2000 IRMA 
International Conference in Anchorage, Alaska Model. 
The IRMA Curriculum [http://www.irma- 
intemational.org/] is a generic framework for an 
international curriculum on information resources 
management in schools of management and is designed 
to be customized to accommodate institutional require- 
ments. Information Systems-Centric Curriculum 
(ISCC’99), funded by the National Science Foundation, 
contains guidelines for educating the next generation of 
information systems specialists and has been developed 
jointly by academicians and practiticHiers from industry 
[http://www.iscc.unomaha.edu/]. The ISCC’99 curri- 
culum is intended to prepare information specialists for 
the development and use of large information systems. 

There are primarily two processes of accreditation that 
impact information systems curriculum. Schools and 



departments of management are concerned with 
achieving and maintaining AACSB [http://www.aacsb 
.edu] accreditation. Second, is the new information 
systems accreditation conducted by the Computing 
Accreditation Commission (CAC) of the Accreditation 
Board for Engineering and Technology (ABET) 
[http://cis.bentley.edu/isa/pages/accreditation.html]. 
The latter accreditation is geared towards the 
approximately 50% of IS programs located outside of 
management and business schools. Accreditation is used 
to differentiate programs and is highlighted in program 
and institutional marketing endeavors. 

The objective of this panel is to explore the impact of 
curriculum models and accreditation on IS under- 
graduate curriculum and degree programs. IS educators 
from Schools of Management, Engineering, and 
Interdisciplinary Programs will present their current and 
planned approaches to the IS curriculum. It is hoped 
that the paths taken by these educators will help others 
navigate through the maze of curriculum development. 
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ABSTRACT 

Many of the Association to Advance Collegiate Schools of Business (AACSB) accredited schools require 
undergraduates MIS majors to take a course in the management of information technology. Over half of these 
schools utilize case studies in the teaching of this course. We believe that case studies are an important vehicle for 
teaching crucial IT management issues, particularly in providing students with a real-world example of 
organizational issues, and that case studies are best taught in an active, collaborative environment. Based upon our 
understanding of collaborative learning and collaborative teaching we propose a procedure for enhancing the 
effectiveness of this active learning methodology, and discuss how this methodology has been implemented. 



COLLABORATIVE LEARNING 

Collaborative learning is defined as a learning process 
emphasizing group or cooperative efforts among faculty 
and students, stressing active participation and 
interaction on the part of both students and instructors 
(Brufee, 1984). Collaborative learning has long been 
stressed as an effective teaching methodology by 
theorists (Vygotsky, 1978). A review of the literature of 
peer/collaborative learning can be found in McKeachie 
(1999). 

The importance of collaborative learning extends to the 
business environment, particularly in the use of teams to 
accomplish business tasks. The effectiveness of using 
teams to accomplish information systems tasks in the 
business environment is well recognized (El-Shinnawy 
and Vinze, 1998; Janz, 1999). Surveys of employers 
indicate that teamwork skills are among the most 
important when evaluating IS graduates for entry-level 
positions. Employers have rated teamwork skills as 
more important than systems analysis and design, 
database, or programmingskills (Van Slyke, Kittner and 
Cheney, 1998). 



Collaborative learning is recognized as an effective 
teaching methodology in MIS programs in the United 
States. Through collaborative learning, students learn to 
take advantage of each team member’s expertise and to 
experience first-hand the problems of coordinating a 
team effort (Goyal, 1995/1996). Studies have shown 
that collaborative learning leads to a higher degree of 
satisfaction with the learning process, to a greater 
motivation to learn, and to better performance (Flynn, 
1992). 

Aram and Noble (1999) argue that the traditional lecture 
approach does not adequately prepare students to 
understand and cope with the levels of ambiguity and 
uncertainty they will inevitably face when assuming 
entry level positions. Collaborative learning can be 
utilized in a number of class settings, and it is 
particularly appropriate for system development 
projects. Collaborative learning can also be effectively 
used for research projects and simulations. However, it 
is our intention to focus on how teachers can utilize 
techniques of collaborative learning in the teaching of 
case studies in the Management of IT course. 
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COLLABORATIVE TEACHING 

Instructors in a collaborative teaching environment can 
also realize some of the same advantages enjoyed in a 
collaborative learning environment. In particular, 
instructors can benefit from synergisticeffects promoted 
by team dynamics. In addition, collaborative teaching 
can lead to a quicker development of best teaching 
practices. 

Brabston, et al. (1999) proposed three models for 
collaborative team teaching: 

1. The interactive model. Two or more instructors in 
front of the class at any one time. 

2. The rotational model. Each member of the teaching 
team teaches in only that part of the course related 
to his or her area of expertise. 

3. The participant-observer model. Each team member 
alternately takes the lead in teaching. The other 
team member primarily observes but also actively 
participates when appropriate. 

Buffington and Harper (2001) have presented a fourth 
model for informal collaborative team teaching. 

CASE STUDIES 

Case studies are an impatant tool for teaching MIS 
concepts (Tracy and Waldfogel, 1997). Romm and 
Pliskin (2000) cite a study by Lee, Trauth andFarwell 
(1995) showing four major clusters of knowledge/skills 
required of MIS personnel in the upcoming decades: 

1. Technical Specialties Knowledge/Skills: including 
operating systems, programming languages, 
database management systems, networks, 
telecommunications, etc. 

2. Technology Management Knowledge/Skills: 
including issues such as where and how to deploy 
information technologies effectively for meeting 
strategic business objectives. 

3. Business Functional Knowledge/Skills: including 
how to re-engineer business processes before the 
adoption of a new information system to produce 
maximum benefit from the system. 



4. Interpersonal and Management Knowledge/Skills: 
which relate to the “boundary-spanning” role of IS 
personnel. This role requires IS professionals to 
master interpersonal skills such as selling, 
negotiating, leading, and counseling. 

Romm and Pliskin (2000) note that of these four skills, 
three of them — technology management, business 
knowledge, and interpersonal skills — are not the 
traditional “hard skills” associated with an IS education, 
but rather can be classified as “soft’ skills because they 
emphasize an understanding and ability to work with 
people rather than machines. Romm and Wong ( 1 997) 
persuasively argue that the best way of teaching these 
soft skills is through the use of case studies. 

Case studies are routinely used in a number of MIS 
courses. In a content analysis of 34 electronic 
commerce course syllabi, Sendall (1999) found that 44 
percent of e-commerce classes were incorporating case 
studies as part of the curriculum. 

Our focus is on the IT management course at the 
undergraduate level. We believe that this course is 
critically important to the MIS major, and we share the 
surprise of O’Hara and Stephens (1999), who found that 
this course is not universally required at AACSB- 
accredited schools. In their study, O’Hara and Stephens 
content analyzed 39 undergraduate syllabi of the IT 
management course. They found that the most common 
assessment method of students in this course were 
exams, quizzes, case study analyses, research papers or 
topic studies, computer-based projects, reports, and 
assignments. Of the 39 courses, only 5 1 percent utilized 
case studies. Further, case analyses accounted for only 
16 percent of the grade, on average. 

TEACHING CASE STUDIES 
THROUGH COLLABORATIVE LEARNING 

Case studies can be taught with many different 
methodologies. Romm and Mahler (1991) describe five 
methodologies: 

1 . Individual processing. Students prepare for cases 
as individuals. 

2. Chronological group discussion. Each case is 
presented via a team (and instructor) with the team 
intact throughout the interactive discussion. 
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3. Simultaneous group discussion. Each case is first 
discussed in sub-units, which later recombine as one 
large group. 

4. Chronological group dramatization. Cases are 
dramatized with the all students serving either as 
actors or audience. 

5. Simultaneous group dramatization. Students first 
break into sub-units which later recombine for case 
dramatization. 

Each of these methodologies has its virtues, and each 
involves a certain amount of active learning. Annette 
Jones (2000) summarizes the argument for active 
learning: 

Active learning is based on the assumption that 
learning is by nature an active undertaking, and 
that different people learn in different ways 
(Meyers & Jones, 1993); it presumes that 
students learn best by doing. Active learning 
provides opportunities for students to talk and 
listen, read, write and reflect on course content 
through problem-solving exercises, small group 
discussions, simulations, case studies and other 
activities. Biggs (1999) also suggests that 
active engagement in the learning process 
encourages the less academic student to employ 
high-level engagement techniques such as 
theorization, reflection, application, whidi are 
more naturally adopted by the more academic 
student even if the teaching method is more 
passive. 

We believe that one of the greatest strengths of teaching 
cases is the flexibility which they provide the instructor. 
A teaching case allows an instructor to choose the level 
of depth for discussion of a topic, as well as which 
topics, theories, and practices are discussed. While 
many teachers have developed their own pedagogical 
methods for teaching cases, there is no generally 
accepted prescription for one “right” way to teach cases. 

As we continue to sharpen our teaching skills, several 
questions occur to those of us that teach cases in class. 
These questions generally center on inquiry as to 
whether our approach is the most appropriate. 
Generally, the purpose of case instruction is to provide 
a real-world example of the issues that organizations 
must face. Such exposure allows students the 
opportunity to identify issues and problems faced by a 



firm, to see vague, conflicting and often ill-structured 
business scenarios, to evaluate decisions made by the 
principles, to relate theory and concepts to a specific 
instance, and/or to make recommendations about what 
shouldbe done based upon the student’sown knowledge 
of the subject matter. As such, it is always our hope that 
the material will “come alive” for the students, 
generating high interest because of the fact that the 
issues are real and the companies are struggling to deal 
with them. 

Teachers have often assigned cases to small groups and 
then have these students present their analysis to the 
class. We term this apjxoach the “traditional case 
approach.” However, some teachers who have been 
teaching cases for a while have developed his or her 
own particular method for conducting discussion of a 
case in class. We next sketch out the two methodologies 
we have most recently followed. 

CASE STUDIES IN THE IT MANAGEMENT 
COURSE AT OUR UNIVERSITY 

Each of the two authors of this study teaches the IT 
Management course. For the past four years. Instructor 
“B” teaches the course in the fall semester, and 
Instructor “A” in the spring. Both of us are proponents 
of active, collaborative learning in the teaching of cases. 
Both of us are interested in improving the effectiveness 
of our teaching. We have in the past informally 
discussed the teaching of cases in our respective classes, 
and have determined that a more formal approach to 
improving our teaching is in order. 

As a result of our informal discussions we have 
developed a case study evaluation instalment. It was 
administered for the first time in spring 2001, and was 
subsequently administered at the end of the fall 2001 
and spring 2002 semesters. Below, we describe the case 
teaching methodologies employed by each instructor in 
his respective section. 

The Spring 2001 IT Management Class — 
Instructor “A” 

Inspired by a workshop hosted by Larry K. Michaelsen 
(see Michaelsen, 1997-1998) prior to the beginning of 
the semester and disappointed by the negative feedback 
I received from my spring 2000 class, I made several 
sweeping changes to my case study methodology. In 
spring 2001 , 1 divided my class of 22 students into seven 
teams. I consciously used principles of demographic 
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diversity as advocated by Trimmer, Van Slyke and 
Cheney [1999] in comprising the teams. I also ensured 
that there would be at least one “high-performing” 
student on each team. The teams endured throughout 
the semester. Although Michaelsen strongly advocates 
giving students all the class time they need to operate in 
groups, I purposefully composed teams whose members 
shared some free time at some point during the week. 
Further, I encouraged teams to conduct e-meetings to 
discuss case questions and to prepare the final report. 

Five case studies were assigned at the beginning of the 
semester, with case discussion to begin on the fifth week 
of the semester. The cases came from Turban (1999). 
Text questions for each of the five cases were 
supplemented with my own questions. 

On each case studies day, the period began with all 
students taking a short multiple-choice quiz on the 
details of the case. As advocated by Michaelsen in his 
workshop, students then immediately grouped together 
in their teams to discuss the quiz and to retake it, this 
time as a group quiz. The quizzes served to motivate 
students to read the case carefully. 

I then led the discussion of the case questions in a 
question-answer format. Students frequently enlivened 
discussion with vigorous debate, as opposing points of 
view were enthusiastically presented. Invariably, dis- 
cussion would fill the remainder of the class period, and 
the following class period was dedicated to tying up the 
loose ends of the case. 

One week after the case discussion concluded, each 
team submitted a written analysis of the case. At the 
end of the semester, all students rated the relative 
contributions of their teammates. Students were forced 
to give at least one teammate more points than the rest, 
a practice advocated by Michaelsen. Altogether, the 
case work (quizzes and reports) was worth 25 percent of 
the course grade. Forcing students to allocate points 
unevenly led to several students having their course 
grade elevated or demoted a level. 

The Fall 2001 IT Management Class — Instructor “B” 

I assign a short, end-of chapter case at least one week in 
advance. I ask the students to read the case and answer 
the questions included in the text after the case. The 
students’ written (wordprocessed) answers are turned in 
to me after the case is discussed in class. I make it very 
clear that their work is not graded in terms or “right” or 



“wrong”; instead, I simply look at each paper to 
determine whether a thoughtful and justifiable response 
has been formulated. Once the cases have been turned 
in, I grade each student’s work by assigning a check 
mark (^) or minus (-) indicating whether I have deemed 
their work to be sufficient. Insufficient answers are 
relatively rare. Those receiving a check mark are given 
credit for all of the points for the assignment, while 
those who receive a minus receive no credit. 

The written answers to the case questions serve as 
reference for the students as we discuss the case in class. 
Because of the availability of some written guidance, 
many of the students seem more at ease when they are 
called upon to contribute to the discussion. Also, I have 
noticed that students seem to be more prone to add to 
the discussion voluntarily when they have a well- 
formulated response in writing at their disposal. 

I purposefully don’t read the questions accompanying 
the case before the class discussion. Instead, I work up 
my own set of questions. My reasoning for this 
approach is that if I read the questions prepared for the 
case, I may actually constrain my own thoughts about 
the issues. I prefer to lead the discussion on what I feel 
is most important to emphasize. Only after I have 
exhausted my own list of questions for the class will I 
ask for responses to the given case questions if, in fact, 
we have not already covered the question in our 
discussion. 

When I have a classroom that allows for rearranging the 
student seating, I ask the students to move their chairs 
into a circle. I also sit in the circle. This arrangement 
seems to improve the informality of the setting and is 
very conducive to group discussion. 

Student comments from previous years of teaching cases 
through this approach, both formally through instructor 
evaluation reports and informally through discussion 
with individuals, have been very positive about the 
value of teaching cases in my classes. The anecdotal 
evidence for the success of the approach is strong but 
indirect. 

The Spring 2002 IT Management Class — 
Instructor “A” 

Buoyed by the success of the spring 2001 approach, I 
made only one important format change for the new 
semester. Because the most negative reaction to the 
approach in 2001 was to the requirement that forced 
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students to rate at least one member’s contributions 
higher than the others, the requirement was dropped. 
Personally, I had serious reservations about the forcing 
requirement. In spring 2002, students did not evaluate 
teammates’ contributions, but were free to drop 
noncontributing member’s names from the written 
reports. 

Another small change is that the case work dropped 
from 25 percent of the course grade to 23 percent. The 
only other apparent difference between the two 
semesters was the increase in class size — from 22 to 3 1 . 
The number of teams increased from seven to ten. 

Instrument Development and Administration 

We developed a 12-item questionnaire administered in 
the spring semester in a collaborative fashion. The 
instrument itself (see Appendix A) is designed primarily 
as an exploratory tool to assess the effectiveness of our 
approach to teaching case studies. We believe a basic 
value of the case approach is in the teaching of soft 
skills (Romm & Pliskin, 2000), which led to the 
development of questions two and six. We also believe 
that the case approach is an excellent vehicle for 
teaching the Bloom’s (1965) higher levels of learning, 
hence questions five, seven, eleven, and twelve. 
Because many of the changes in approach were inspired 
by Michaelsen (1997-1998), we developed questions 
four, eight, and nine. Question three was developed to 
assess students’ perceptions of cases as agents of active 
learning, as suggested by Horgan (1999). Both 
instructors felt that cases were an important tool for 
teaching key issues, resulting in question one. Finally, 
we wanted to learn whether students preferred our new 
approach to the more traditional case approach, which 
accounts for question ten. The instrument concludes 
with two open-ended questions to explore issues not 
sampled by the first twelve questions. 

Students anonymously evaluated the case studies at the 
end of each semester. A five-point Likert scale was 
used to evaluate the first twelve questionnaire items. A 
total of 22 students participated in the spring 2001 
survey; 20, in fall 2001; and 26 in spring 2002. Because 
of the differences in teaching styles between the two 
instructors, some items were not applicable to Instructor 
“B’s” class, specifically items 4, 6, 8, and 9. The results 
of this survey from each semester are shown in the 
tables below. Questions have been sorted from their 
original arrangement, to an order showing statements 
with which students most strongly agreed first. 



Results 

The results of the spring 2001 survey (Table 1) indicate 
a widespread satisfaction with the approach to case 
studies. Our perception is that students like this case 
methodology much more than they did in the spring 
2000 class. Students seem to be particularly satisfied 
with cases as tools for making abstract MIS principles 
concrete. Not surprisingly, the question which received 
the least support concerned the forcing of uneven 
ratings — which led to some students receiving lower 
grades for the course than they otherwise would have. 
However, only two of the twenty-two students either 
disagreed or strongly disagreed with question nine. 

Even the response to question ten was gratifying. This 
question, which ranked tenth in order of agreement, 
asked students to compare the spring 2001 approach 
with the traditional approach. Although three students 
rated the question neutral, not one of the twenty-two 
either disagreed or strongly disagreed with the 
statement. 

In addition to the twelve questions above, students were 
also asked to respond to these two open ended questions: 

1. What is the one thing you liked best about our 
approach to cases this semesto? 

2. If you could change one thing about our approach to 
cases this semester, what would it be? How would 
you change it? 

As might be expected, a number of students rephrased 
one or more of the twelve statements to indicate the 
greatest strength. As proponents of active learning, we 
are pleased to note that six students indicated that active 
class discussions were the greatest strength of this 
approach to case studies. 

Even responses to the “greatest weakness” question 
tended to be positive. In fact, five students indicated 
that there was no weakness with the 2001 methodology. 
Typical responses to this question included: 

• More than five cases should be used (2) 

• More in-class time should be allocated to cases (2) 

• Class should be 75-minutes rather than 50 minutes 

Complaints tended to be about the cases themselves 
rather than to the methodology: 
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TABLE 1 

RESULTS OF CASE STUDIES QUESTIONNAIRE 



Question 

Number Question Mean 

7. The cases provide students with a good means of applying inform ation system s principles to real world 1 .36 

situations. 

1. The cases broughtout importantpoints about managing information systems, such as the role of IS in 1.41 

a global economy, the potential of e-commerce, the role of IT in strategic planning, IT ethics, etc. 

1 1 . The cases provide students with a good opportunity to synthesize; that is, identifying potential solutions 1 .45 

to a case problem and choosing the most appropriate solution. 

12. The cases provide students with a good opportunity to exercise evaluation skills, i.e., appraising the 1.45 

extent to which particulars are accurate, effective, economic, or satisfying. 

4. Having both an individual quiz and a group quiz is a good idea. 1.55 

2. The cases are a good way of teaching “soft skills”; for example, interpersonal skills and management 1.64 

skills. 

3. Cases increase the likelihood of student participation in class discussion. 1 .64 

6. Because much of the case work involved team work, the cases served as a good vehicle for applying 1.68 

principles o f team m anagem ent. 

5. Writing the case report aided in understanding the case principles. 1.73 

10. I prefer the approach to cases used this semester to the traditional case approach, i.e., when studentteams 1 .73 

are assigned the responsibility of presenting a particular case. 

8. Requiring students to assess relative contributions of teammates is a good way to motivate individual 1.77 

efforts. 

9. Requiring students to rate at least one teammate’s contribution as better than average is a good idea. 2.41 



TABLE 2 

RESULTS OF CASE STUDIES QUESTIONNAIRE— FALL 2001 



Question 

Number Question Mean 

3. Cases increase the likelihood of student participation in class discussion. 1 .65 

7. The cases prov ide studen ts with a go od me ans of ap plying in forma tion system s principles to real world 1.75 

situations. 

1. The cases broughtout importantpoints about managing information systems, such as the role ofIS in 1.85 

a global economy, the potential of e-commerce, the role of IT in strategic planning, IT ethics, etc. 

5. Writing the case report aided in understanding the case principles. 1.85 

1 1 . The cases provide students with a good opportunity to synthesize; that is, identifying potentialsolutions 1 .85 

to a case problem and choosing the most appropriate solution. 

12. The cases provide students with a good opportunity to exercise evaluation skills, i.e., appraising the 2.00 

extent to which particulars are accurate, effective, economic, or satisfying. 

10. I prefer the approach to cases used this semester to thetraditional case approach, i.e., when studentteams 2. 10 

are assigned the responsibility of presenting a particular case. 

2. The cases are a good way of teaching “so ft skills”; for example, interpersonal skills and management 2. 1 5 

skills. 

6. Because much of the casework involved teamwork, the cases served as a good vehicle for applying N/A 

principles o f team m anagem ent. 

4. Having both an individual quiz and a group quiz is a good idea. N/A 

8. Requiring students to assess relative contributions of teammates is a good way to motivate individual N/A 

efforts. 

9. Requiring students to rate at least one teammate’s contribution as better than average is a good idea. N/A 
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BEST COPY AVAILABLE 



• Cases should be more current (2) 

• Cases should be more detailed 

The open-ended responses tended to be similar to those 
of the Spring 2001 class responses. The responses to 
“What is the one thing you liked best about our 
approach to cases this semester?” included the dynamics 
of increased class discussion and the “real world” 
application of principles and theory. We note that these 
responses correlate to the two highest-scoring items in 
the questionnaire. 

There was very little response to the second open-ended 
question, “If you could change one thing about our 
approach to cases this semester, what would it be? How 
would you change it?” Most students chose not to 
respond to this question or simply replied “Nothing.” 
One student stated that there should be more cases. 
Another student stated that additional in-class time was 
needed to adequately discuss the cases. 

A comparison between the spring 2001 and spring 2002 
survey reveals that student attitudes are generally the 
same — students have a favorable reaction to this 
approach to using case studies. The responses to the 



open ended questions again were mostly favorable. 
Nine of the twenty-six students indicated that what they 
liked best about the approach was the class discussion. 
Another five indicated that the “groq? work” was the 
one thing they liked best. 

There were no noticeable or consistent patterns to the 
responses to the question 14 — what would you change. 
Indeed, five students omitted a response to this question. 
Three students indicated that the written reports were 
redundant, given the considerable in-class discussion. 
This conclusion could be attributed more to students’ 
desire to lighten their work load than to actual increase 
in effectiveness. Two students suggested that we have 
more than five cases, and two suggested having fewer 
than five cases. Two students suggested more detailed 
cases and two suggested spreading the cases more 
evenly throughout the semester. 

However, there are three differences between tables one 
and three that are worthy of note. Question 10 receives 
about the same level of favorable support in both 
semesters. However, the question moves from tenth 
position to fifth position in it rank order. Question 
number twelve shows the greatest change in mean 



TABLE 3 

RESULTS OF CASE STUDIES QUESTIONNAIRE— SPRING 2002 



Question 

Number Question Mean 

1 . The cases brought out important points about managing information systems, such as the role of IS 1.73 

in a global economy, the potential of e-commerce, the role of IT in strategic planning, IT ethics, etc. 

7. The cases provide students with a good means of applying information systems principles to real 1.81 

world situations. 

1 1 . The cases provide students with a good opportunity to synthesize; that is, identifying potential 1 .96 

solutions to a case problem and choosing the most appropriate solution. 

4. Having both an individual quiz and a group quiz is a good idea. 2.00 

10. I prefer the approach to cases used this semester to the traditional case approach, i.e., when student 2.00 

teams are assigned the responsibility of presenting a particular case. 

6. Because much of the case work involved team work, thecases served as a good vehicle for applying 2.08 

principles o f team management. 

3. Cases increase the likelihood of student participation in class discussion. 2.12 

2. The cases are a good way of teaching “soft skills*’; for example, interpersonal skills and 2.1 5 

management skills. 

8. Requiring students to assess relative contributions of teammates is a good way to motivate 2.15 

individual efforts. 

12. The cases provide students with a good opportunity to exercise evaluation skills, i.e., appraising the 2.15 

extent to which particulars are accurate, effective, economic, or satisfying. 

5. Writing the case report aided in understanding the case principles. 2.19 

9. Requiring students to rate at least one teammate’s contribution as better than average would be a 2.89 

good idea. 
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(0.70), moving from third to tenth. In all other cases, 
rank order is virtually identical. 

The most striking difference between tables one and 
three is the uniform decline in favorability. In all cases, 
the level of satisfaction declined an average of 0.45, 
nearly half a point, ranging from a decline of 0.27 to 

0. 70. There are at least three factors which can account 
for this decline. 

The first factor is that many of the students in the spring 
2002 class, sixteen of the thirty-one, were also in my 
decision support systems class in fall 2001. I received 
the second lowest evaluations in my 17-year teaching 
career in this class. I believe that there was some carry- 
over from this relatively negative experience in fall 
2001 to the spring 2002 class. 

A second factor is the increase in class size from 22 to 
31, a forty-one pa*cent increase. As Mostert and 
Sudzine have shown, larger class sizes have a negative 
effect on case discussions. The size of the classroom 
did not change, and the ten groups were not able to meet 
without rubbing elbows with other members of the 
group. 

I believe the most important factor, however, has to do 
with investment. In spring 2001, I was full of 
enthusiasm and energy with the new approach. I spent 
an entire class period early on in the semester describing 
how the new and innovative approaches we were taking 
to cases. Students eagerly provided input to fine-tuning 
the new approach. In spring 2002, however, I merely 
announced how we were approaching cases. Students 
did not feel they had the same investment in the 
approach as the previous semester. 

LESSONS LEARNED 

While Brabston, et al. (1999) proposed the three models 
for collaborative team teaching as described earlier in 
this paper, we would like to propose a fourth, less 
radical model. We suggpst a more formal approach to 
procedures that good instructors are already doing 
informally. We propose that when two or more 
instructors are assigned responsibility for the teaching of 
a particular class using case studies as a teaching tool, 
that these instructors routinely perform the following 
steps: 

1. Compile a list of generally agreed upon desired 
outcomes from teaching the cases. 



2. Construct a questionnaire designed to evaluate the 
processes used to achieve the outcomes. 

3. Administer the questionnaire at the conclusion of 
each semester. 

4. Meet to discuss questionnaire results; identify 
methodologies that best meet desired outcomes. 

5. Incorporate appropriate methodologies in future 
classes. 

We believe that, while no “one-size-fits-all” when it 
comes to teaching methodologies, this process can only 
result in more effective teaching. A structured approach 
to evaluating methodologies from multiple instructors 
teaching a common course, as we have begun to do, 
should lead to a fme-tuning of individual teaching 
performance. 

The case study format presents an excellent opportunity 
for instructors to collaborate in the determination of 
which methods and desired outcomes are most 
appropriate for a course. Based on the results of the 
spring 2002 questionnaire. Instructor “A” plans to 
reinstitute the policy of having students evaluate team 
members’ performances, but not to force one member to 
be rated above the others. Additionally, Instructor “A” 
plans to enthusiastically sell students on this case 
approach at the beginning of the semester. 

We give here one more example of our dynamic, 
collaborative methodology. Question three was ranked 
highest in terms of respondent agreement in the class 
taught by Instructor “B.” The same question ranked 
seventh in both classes taught by Instructor “A.” A 
possible reason for this difference may be that Instructor 
B’s teaching style is less likely to promote discussion in 
class. If this is the case. Instructor B’s students may 
have welcomed the opportunity to participate more 
broadly in discussion when the case studies were being 
taught, thus recognizing this important value of case 
teaching methodology. 

In this paper, we have presented a combination of 
anecdotal and empirical evidence of the benefits 
instructors (and, ultimately our students) can derive 
from the above model. We believe this process can only 
result in more effective teaching. We believe that there 
is no one-size-fits-all teaching methodology, but we do 
believe a structured approach to evaluating our 
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methodologies, as we have begun to do, should lead to 
a fine-tuning of individual teaching performance. 

Particularly given the AACSB’s emphasis on 
assessment (Williams and Price, 2000), we believe that 
our model of collaborative teaching should serve as an 
excellent springboard for improving instruction. The 
responses to questionnaire items are valuable, but the 
discussion they provoke is invaluable. We do not 
believe it is enough to simply divide students into 
groups to discuss case studies. The devil is in the 
details. 

For example, if many students indicate that “lively 
discussion” is a great strength of one instructor’s 
approach, the positive response is not necessarily due to 
the content of the questions. Discussion of this item 
may reveal that lively student participation is a product 
of the instructor’s serving more as a facilitator than as a 
leader. Discussion of these and other questions lead to 
synergistic improvements. Good teaching is certainly as 
much art as science, but it is an art enhanced by our 
collaborative procedure. 

LIMITATIONS 

The results of this questionnaire must be interpreted 
with caution. The instrument used in this study was 
constructed with elements which seemed relevant at the 
time. We plan to revise the instrument by grounding it 
more firmly in the literature and by following with 
validation tests. Another limitation is that apparent 
student satisfaction with the 200 1 methodology does not 
ensure that the methodology was more effective. There 
could have been contaminating factors — the students 
may have found that pioneering a new methodology was 
a positive experience. Or the instructor may have done 
a more effective job for all aspects of the 
course — pulling the case study results up by the 
bootstraps, as it were. Nevertheless, the evidence does 
point to our collaborative case approach being an 
effective way of teaching cases. 
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APPENDIX A 



Case Questionnaire Name: 

Use the scan tron to answer the first twelve questions below. Do not put your name on the scan tron. Y ou must write 
your name above in order to receive the ten points extra credit. 

A: strongly agree B: agree C: neutral D. disagree E. strongly disagree 

1. The cases brought out important points about managing information systems, such as the role of IS in a 
global economy, the potential of e-commerce, the role of IT in strategic planning, IT ethics, etc. 

2. The cases are a good way of teaching “soft skills”; for example, interpersonal skills and management skills. 

3. Cases increase the likelihood of student participation in class discussion. 

4. Having both an individual quiz and a group quiz is a good idea. 

5. Writing the case report aided in understanding the case principles. 

6. Because much of the case work involved team work, the cases served as a good vehicle for applying 
principles of team management. 

7. The cases provide students with a good means of applying information systems principles to real world 
situations. 

8. Requiring students to assess relative contributions of teammates is a good way to motivate individual efforts. 

9. Requiring students to rate at least one teammate’s contribution as better than average is a good idea. [Note: 
questions was reworded as “Requiring students to rate at least one teammate’s contribution as betta than 
average would be a good idea” in the spring 2002 survey.] 

10. I prefer the approach to cases used this semester to the traditional case approach, i.e., when student teams 
are assigned the responsibility of presenting a particular case. 

11. The cases provide students with a good opportunity to synthesize; that is, identifying potential solutions to 
a case problem and choosing the most appropriate solution. 

12. The cases provide students with a good opportunity to exercise evaluation skills, i.e., appraising the extent 
to which particulars are accurate, effective, economic, oc satisfying. 

13. What is the one thing you liked best about our approach to cases this semester? 

14. If you could change one thingabout our approach to cases this semester, what would it be? How would you 
change it? 
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ABSTRACT 

One of the critical components in the curriculum is evaluation of the student's knowledge or the amount they learned 
in the course. Overtime, the assessment process has not changed significantly; it is only recently that university 
instructors are turning to technology for assistance in this labor-intensive task. There are disadvantages and 
advantages for both instructors and students of on-line testing. Much of the discussion surrounding alternative 
testing implementation centers on various learning styles. One question is ''why do we continue to assess students 
in traditions set long ago? ” Some other questions are raised: Does the on-line testing improve student learning and 
achievement? Can/should instructors tailor tests to student's preferences for learning? How does the creation of 
on-line exams affect the instructor's work-load? This research looks at two traditional face-to-face Information 
Systems Courses which incorporated on-line testing into their curriculum and begins to provide some answers to 
these questions. 



INTRODUCTION 

“On-line education can be defined as teaching and 
learning activities enabled by electronic media” (Romm 
and Taylor, 2000). Electronic media or “technology is 
not simply an add-on service as computers or 
audiovisual...” (Tsichritzis 1999, 93). Emerging 
technologies are an integral part of the university 
curriculum — not only are they, themselves, studied, but 
also the manner they sipport the delivery of many facets 
of the curriculum. One of the critical components in the 
curriculum is evaluation of the student’s knowledge or 
the amount they learned in the course. Over time, the 
assessment process has not changed significantly (Natal, 
1998); it is only recently that university instructors are 
turning to technology for assistance in this labor- 
intensive task. 

There is a growing emphasis on accountability offaculty 
and an increased importance on ensuring student 



success, where success is being defined as student 
learning. AACSB reviewers are evaluating business 
curriculum with a strong interest in outcomes assess- 
ment — measures of student learning. The questions 
being asked are- are the students learning what the 
university thinks they are learning and can it be 
measured? How is improvement in student learning 
measured? Is the testing environment secure? The 
answers to these questions are being confounded by the 
proliferation of distance education or on-line courses 
using the Internet as a means of delivery. 

For a course that is taught fully on-line or distance, it is 
expected that the testing component is also 
accomplished at a distance or on-line. However, on-line 
testing or assessment may be an appropriate venue for 
courses taught in a face-to-face environment. It may 
relieve part of the repetitious, time-consuming activities 
involved with creating exams, grading exams, and 
calculating the exam scores and final grades. More time 
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might be spent in the actual teaching of the course 
content and less on administering exams. This research 
looks at incorporating on-line testing into a traditional 
lecture-based course. 

BACKGROUND 

Most of the on-line testing literature involves training or 
use of computer-based instruction (CBI)/computer- 
based training (CBT) (Woit and Mason, 2000; Tunc and 
Armstead (2001); Kaczmarczyk 2001) and the usage is 
either in the corporate workplace or the elementary/high 
school skills-based context. Recently, with the new 
emphasis on technology, increased band-width and the 
dominance of the Internet, university professors are 
incorporating some of this technology into their courses. 
On-line testing is easy to track and document (Roberts 
2000; Tunc and Armstead 200 1 ; Woit and Mason 2000) 
and can be used to establish baselines for future 
assessment. Feedback mechanisms can be incorporated 
into the on-line testing (Gibson, Tesone, Blackwell 
200 1 ) and different learning styles of the students can be 
accommodated. Therefore, this mode and process of 
testing or assessment may be more effective and 
efficient for both faculty and students. 

Dottie Natal (1998, 7-9) of Imagen Multimedia 
Corporation presented the advantages and disadvantages 
for both instructors and students of on-line testing at the 
Technology Education Conference in Santa Clara. 
Advantages or benefits for students are: 

• “Any time, any place” testing 

• Support for ESL (English is a second language) 
students 

• Support for students with disabilities 

• Valuable class time not used for exams 

• Depending upon the type of exam — immediate 
feedback may be available 

• Access to tools they are comfortable with 

Drawbacks for student include: 

• Procrastination - last minute test taking 

• Availability of computers 

• Lack of computer skills 

• Inability to use their standard test-taking skills 
(answering the hardest first, sure answers first....) 

Some benefits for the instructor are: 

• Decreased record-keeping time 



• Reusable test questions 

• Easier reading than hand-written 

• Increased teaching time 

• Better test design — for differait learners 

• More frequent assessments 

• Better statistical feedback 

Drawbacks for instructors are: 

• Security problems 

• Unexpected results with regard to writing standards 

• Debugging problems with feedback or test 
administration 

• Hardware problems 

Other research has provided additional instructor 
enhancements to include immediate assessment, there- 
fore, enabling the instructor to review areas particularly 
challenging for individual students based upon on-line 
testing scores. This in turn provides class time for 
discussion of more difficult topics, instead of 
administering quizzes and exams (Woit and Mason 
2000; Tunc and Armstead 2001; Kaczmarczyk 2001). 

On-line testing has been identified with distance 
education courses. These are courses in which students 
enrolled do not attend class, do not accomplish the 
related tasks within a set time frame and can be at any 
location. Many of these students have computers at 
home, or have access at the workplace and have 
mastered the software. Why not use this mode of 
assessment for students enrolled in traditional face-to- 
face university courses? 

Universities provide access to computer labs throughout 
campus and in the domiitories. Some universities 
require computer ownership for admission. Universities 
are providing wireless connections to email and the 
Internet enabling laptops to be used anywhere on 
campus. In one scenario, on-line testing might be 
administered in a controlled setting, with-in one of the 
computer labs. 

One university experimented with installing a fully- 
outfitted computer lab for quiz taking. A proctor was 
hired to monitor testing activities. Students reserved a 
computer for completing the required quiz (Tunc and 
Armstead 2001). However, there is always the danger 
of the system being unable to accommodate users 
accessing the system at exactly the same time; each 
system has its own saturation point which must be taken 
into account. 
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Another scenario might have all students taking the test 
at the same time, but from any location they choose. 
This strategy not only raises overioading-the-system 
issue, but also monitoring the students \\hile they are 
taking the test. Are they using books, documents, the 
Internet? Does the instructor care as long as the student 
is learning while they are completing the exam? Who is 
really taking the test? Will the technology create new 
methods of cheating? 

A third scenario is any time, any place, without 
restraints. This probably eliminates the system-overload 
issue, but not the other issues. Which delivery mode is 
optimal depends upon the preferences of the faculty and 
students and the university’s level of technology and 
support for that technology. 

LEARNING STYLES 

Much of the discussion surrounding alternative testing 
implementation centers around recent readings of 
various learning styles and their impact on testing and 
test results. Considerable research exists in tacit and 
explicit learning that will not be repeated here. 
Nevertheless, profound research conducted by Howard 
Gardner, author oi Frames ofMind{\9%'i), which led to 
further research, and issuing of Multiple Intelligences 
(1993^ identified educators’ oversight of various forms 
of intelligence that exists. 

Dr. Gardner’s work identifies more than one form of 
intelligence. Referred to as multiple intelligences, seven 
varieties of intelligence are identified by Dr. Gardner 
and his team of researchers. During the more than thirty 
years of work, psychologists working with all levels of 
education found that intelligence manifests itself in 
intrapersonal, interpersonal, spatial, linguistic, logical- 
mathematical, bodily-kinesthetic, and musical 
intelligence. It is stated that all individuals may possess 
more than one of the intelligences, and some, typically 
those who are mentally challenged, posses possibly just 
one of the identified intelligences. These multiple 
intelligences provide for a variety of per^ectives on 
how information is learned. However, educators may 
fail to test more than one of these intelligences. 

Currently, assessment of learning is conducted towards 
the logical-mathematical intelligence as specified in 
Multiple Intelligences (1993). This assessment style is 
derives from Binet-Spearman formulation of evaluation. 
In the Binet-Spearman model, “the individual is tested 
in isolation” (Gardner 1983, 52). This approach is used 



when a student sits at a desk during a designated time 
period (that may not be the original class meeting time 
or regular class meeting room) and regurgitates data 
pertinent to the course. As Gardner states “the mind is 
a multifaceted, multi-component instrument, which 
cannot in any legitimate way be captured in a single 
paper and pencil-style instrument” (1993, 70) and 
further that “members of Mensa are e;q)ert in nothing — 
except in taking tests of intelligence” (1993, 53). These 
claims appear to be supported by very successful 
students who may not do well on exams; they were not 
the best test takes. 

The question that begs to be answered is ‘why do we 
continue to assess students in traditions set long ago?’ 
There is a greater understanding of learning styles which 
is not extended to the ability to adequately assess 
student grasp of information. 

ON-LINE TESTING IMPLEMENTED 

On-line testing was implemented in two different 
traditional fact-to-face courses. In one course. Business 
Data Communications, weekly quizzes forced students 
to read the text, become well prepared for class 
discussion and stay up-to-date with the course materials. 
In the other course. Management Information Systems 
Technology (MIST), an on-line test was used for the 
midterm with one section, while the other section took 
the exam in class. Students in both MIST sections were 
given the option for an on-line final exam or an in-class 
final exam. 

Both instructors used an on-line. Internet-based course 
management system. The system is used throughout the 
university in many different types of courses. All the 
students in the Data Communications course had used 
the software in at least one previous course. It was new 
for some students in the MIST course. The software 
provides automatic grading of certain types of questions 
and automatic posting of grades. Faculty can design the 
exam in any format. The easiest design may be 
true/false and multiple choice, but essay questions are 
accommodated. Immediate feedback on an individual 
answer is not a feature of the software; after the test is 
completed the student receives feedback on the entire 
exam, the software generates a comparison page of 
student answers and correct answers. 

The data communications course is an elective in the 
Information Systems field, the students are Juniors or 
Seniors, and all of the students have reasonable, if not 
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excellent computer skills and should all be familiar with 
the on-line course management program used for 
quizzes. In this course, valuable face-to-face in-class 
time was not spent on the weekly quizzes. The 
professor prepared only true/false and multiple choice 
questions based on the text material. The exam 
preparation time is equivalent to that for an on-line 
exam consisting of the same types of questions. The 
real time savings is in the auto-grading feature of the 
software. Exams were posted each week and the 
students had the entire week to complete the exam. The 
students had one hour to complete the exam; the 
software keeps track of the length of time spent on-line 
in the testing area. During class, following each exam, 
time was spent discussing/arguing about the exam 
questions, thereby reinforcing important concepts. 
Since extra points were earned if their arguments were 
successful, this strategy was a sure way to get students 
to participate in class discussions and stay current with 
the reading material. The students provided feedback on 
their experience. 

• The quizzes were a good learning tool and should be 
expanded 

• The quizzes forced me to stay current with the 
readings 

• The quizzes formed me to focus on the materials and 
class — usually I do not read the book before studying 
for mid-terms 

• I learned more because of the weekly quizzes 

One student commented that because the quizzes were 
administered on-line there was no reason to attend class. 

The MIST course is an introductory core Information 
Systems course and most of the students are 
Sophomores. Many of the students have used the on- 
line course management system before entering the 
course, some have not. However, all requirements and 
assignments for the course are available only on the on- 
line system and by the mid-term, each student should 
have experience with the system. The same instructor 
taught two sections of the MIST course; one section 
completed the mid-term during class time and the other 
section used the on-line system. The on-line students 
were on a limited time schedule, similar to those in the 
classroom. There is a built-in utility in the on-line 
system that releases the test materials at a certain time 
and then makes it unavailable after an elapsed period, 



however, the students were in the location of their 
choosing. Therefore, both sets of student had the same 
amount of time for completing the exam. The tests 
consisted of fifty multiple choice questions and nineteen 
short answer questions. The exam covered all materials 
up to the time of assessment. Both sets of students were 
advised to use textbooks, lecture notes, class papers and 
classmates in formulating their response. Surprisingly, 
exam results were slightly higher for the in-class test 
takers than for the on-line test takers (Table 1). 



TABLE 1 

GRADE FREQUENCIES 





In-class: 
N = 29 


On-line: 
N = 31 


Mean 


97.7 


95.8 


Median: 


98.4 


96 


Mode: 


98.4 


98 


Std Dev. 


1.6 


2.36 


Min/Max 


93.6/100 


88/98 



There were 31 on-line test takers in one section of the 
undergraduate course and twenty-nine test takers in the 
in-class test-takers. Both sections were given the same 
test-taking directions. In-class test takers were asked to 
leave the room if they wanted to discuss the exam with 
classmates. Both exam periods were limited to the 
university requirements of 1 hour and fifty minutes. 

Both sections had the option to take the exam as either 
in-class, on-line, or a mix. Interestingly, students in one 
section chose to take the exam in-class, and the second 
section chose the on-line presentation. In neither section 
had students participated in an on-line testing 
experience. 

Data are limited to test scores. Demographic data on 
students was not collected in either testing environment. 
Nor was the amount of time taken for completion of the 
exam registered. While demographic and completion 
time data is available for the on-line students, these data 
were not collected for students completing the exam in- 
class. 

Comments from students taking the on-line test were 
overwhelmingly positive. Only two mishaps occurred. 
Both were satisfied by reinitiating the exam on-line for 
one student and providing a paper copy for the second 
student. The student opting to complete the exam on- 
line was able to answer all questions in the remaining 
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allotted time period. Since the questions were answered 
once already, the student had no trouble retaking the 
exam. The second student experienced no difficulties in 
completing the written exam. 

When final exam time arrived, both sections were again 
given the option to take the exam in-class, on-line, or a 
mix. The original in-class section opted for a mix of in- 
class and on-line with a majority taking the exam in- 
class. About one-third of students opted to complete the 
exam on-line. The original on-line class resoundingly 
opted for completing the final exam on-line. Part of this 
motivation may be a result of the exam scheduled on the 
very last exam date. This date may have interfered with 
summer plans. Taking the exam on-line allowed 
students to be at any location comfortable to the student. 

DISCUSSION 

On-line testing may add another level of complexity to 
the test-taking process. Either the instructor or the 
students may perceive on-line testing as another obstacle 
to conquer; in the students’ mind it might be another 
deterrent to a good course grade. The level of technical 
skills of the instructor and their confidence in that skill 
are important factors in determining whether an attempt 
is made to incorporate any technology into the 
curriculum. The technical skill level and confidence of 
the student are also important factors. 

Does the on-line testing improve student learning and 
achievement? There was no statistically significant 
difference in the means for on-line exams and in-class 
exams. Additionally, the means for the on-line exam 
were lower than the means for the in-class exam. 
Therefore, in this study, there is no improvement in 
student learning and achievement when an on-line exam 
was administered. 

Can/should instructors tailor tests to student’s 
preferences for learning? Can/should instructors offer 
several versions of the test and let the students select 
their preference — before they see the questions? This 
may give the students more control over their outcomes. 
This issue requires further study and research. 

Another consideration of students in determining 
whether to take the test in-class or on-line may be the 
accessibility of the instructor to answer questions. In 
both instances, students were assured of instructor 
availability for questions arising during the exam. In the 
instance of the mixed test takers (some in-class, some 



on-line) the instructor carried a cell phone set to vibrate. 
In the instance of all on-line test takers, the instructor 
remained in the office during the entire testing period 
thereby providing access through email or telephone. 

How does this affect the instructor’s workload? 
“Faculty may wish to practice on the actual systems the 
students will be using” (Loeding and Wynn 1999, 181). 
A dry run enables the faculty to anticipate problems 
with the system, the software or the exam itself Faculty 
administering on-line testing may have to be more 
flexible than those in a traditional classroom situation. 
Technical difficulties may negatively influence course 
evaluations at the end of the semester and can impact the 
perception of the entire course. In the two instances 
where the on-line exam failed, the instructor was 
flexible in allowing the students to retake the test in the 
on-line environment or to take the exam in a classroom 
environment. Although the task of grading may be 
eliminated for certain types of exams, this flexibility and 
additional access may increase the faculty’s workload. 

The concern about students copying or cheating is 
always present. Who is taking the exam and with whom 
are they taking it? The concept of cheating has long 
been a concern where there is no one monitoring the 
exam. Dr. Jerry Harvey (1988) claims that there is ‘no 
such thing as cheating.’ What Dr. Harvey referred to is 
the realization that learning occurs in various forms 
inside and outside the classroom. Dr. Harvey believes 
that collaboration is instrumental in the leaning process; 
it facilitates and enhances learning. He recognizes this 
need to accommodate various learning styles as an 
extension of future requirements in the chapter 
“Encouraging Future Managers to Cheat” in The Abilene 
Paradox (1988). In this chapter. Dr. Harvey acknow- 
ledges that once students graduate they will not work 
alone but with teams of people. He identifies areas of 
benefit in allowing students to work together in 
completing assessments as “providing a model of how 
work really gets done” (Harvey 1988, 124) and to do 
otherwise “thwarts the expression of synergy” (Harvey 
1988, 125). By providing those skills for which the 
student is best able to provide for the success of the 
team, the concept of conducting on-line testing may 
support positive interaction and building of relationships 
towards conclusion of a task. 

If Dr. Harvey’s claims are accepted, cheating is one less 
item of concern. Furthermore, as documented by 
Kaczmarczyk (2001) cheating as a phenomena during 
on-line exams has not been sufficiently documented. 
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Therefore, instances of cheating in distance or non- 
monitored test taking may not be an issue. In the MIST 
exam scenario, the instructor encouraged bothclasses to 
use class notes, lecture slides, and each other for 
formulating responses to questions. The reality is 
probably time constraints inhibited much if any 
conversation. 

Thunderstorms and power outages can adversely affect 
transmission of the test. Technical problems may make 
it impossible for a student to complete the exam in a 
timely fashion. If the exam is in a controlled computer 
laboratory setting, the system may not be able to handle 
20 to 30 attempts to access it at the same time. If the 
exam is truly an “anyplace” exam, bandwidth may 
influence the assessment outcomes. Students must be 
very knowledgeable of the computer system they are 
choosing. 

On-line testing should only be one component of the 
evaluation of the student (Gibson, Tesone, Blackwell 
2001). Other activities used to asses learning outcomes 
might be assignments, participation credit, projects, 
papers or case studies. 
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ABSTRACT 

Most information system development projects are conducted by a group of people including systems analysts, 
designers, programmers, etc. This paper examines the effects of group efficacy and team building on group cohesion 
and the impact of the later on group performance in systems development projects. A survey was conducted among 
4th-yearMIS undergraduate students workingat real-life system development projects for four months. One hundred 
and eighty five students forming 35 groups participated in this study. The research model draws on small group 
research theory, which asserts the importance of the determinants of group performance to both researchers and 
practitioners. The data analysis method used was Partial Least Squares (PLS). 



INTRODUCTION 

Group research indicates that understanding the deter- 
minants of group performance is a key for modeling of 
team effectiveness. Group work becomes a norm in 
most projects and a growing number of organizations 
are latching on to anything that makes groups perform 
better. As an example, organizations are looking for 
products such as groupware software and web portals 
that allow for online collaborative work to help them 
think faster and get products to market sooner and with 
less expense. In addition, information systems 
development projects often involve systems analysts 
working in groups either face-to-face or through an 
electronic collaborative setting. Hence, organizations 
need to have an in-depth sense of what makes groups 
perform better regardless of the medium of 
communication used. The goal of this study is to 
identify what determine group performance in IS 
development projects. 



THEORETICAL BACKGROUND 
Group Cohesion 

Group cohesion conceptualization and measurement 
goes back to Festinger (1950) who defined it as 

the total field of forces which act on members 
to remain in the group. These forces may 
depend on the attractiveness or unattractiveness 
of either the prestige of the group, members of 
the group, or the activities in which the group 
engages (p. 274). 

While the importance of this construct has been 
identified as a key in effective work groups (Cohen and 
Bailey, 1997), its conceptualization and measurement 
has been centered on one factor structure construct 
(Mullen and Copper, 1994), namely interpersonal 
attraction to the group. Recently, Cota et al., (1995) 
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have called for defining group cohesion as a 
multidimensional construct based on the Group 
Environment Questionnaire (GEQ) developed for sport 
teams by Widmeyer et al., (1985). The authors 
suggested that group cohesion have four dimensions; (1) 
Group Integration-Task, which is an individual team 
member's perception about the similarity and closeness 
within the team about accomplishing the task, (2) Group 
Integration-Social, which reflects individual team 
member’s perception about closeness and bonding 
regarding the team’s social activities, (3) Individual 
Attraction to Group-Task, which describes individual 
team member's feeling about personal involvement in 
the group task, (4) Individual Attraction to Group- 
Social, which reflects individual team members’ 
feelings about personal involvement in the social 
interaction of the group. 

The most important shift in the literature was separating 
task and social cohesion when defining group cdiesion. 
Carless and De Paola (2000) tested the construct as 1) a 
single-factor model, defining cohesion with highly 
interrelated perceptions about the group; 2) a two-factor 
model, based on Widmeyer et al., (1985) which 
distinguishes between individual attraction to the group 
and group integration; 3) a two-factor model, based on 
task and social cohesion, and 4) a four-factor model 
distinguishing group integration-task, group integration- 
social, individual attraction to group-task and individual 
attraction to group-social. The authors found evidence 
that group cohesion construct is multidimensional and 
includes task cohesion (the extent of motivation towards 
achieving the organization’s goals and objectives 
(Widmeyer et al., 1985, p. 17), social cohesion 
(motivation to develop and maintain social relationships 
within the group) and finally individual attraction to the 
group. The present study adopts this conceptualization. 

Group Efficacy 

Group efficacy has been defined as the group’s 
collective estimate of its ability to perform a task 
(Gibson, 1999). Indeed, group efficacy has been 
investigated as to outcomes such as problem solving 
(Cervone and Peake, 1986), learning and achievement 
(Campbell and Hackett, 1986) and traning effectiveness 
(Earley, 1994). Moreover, Bandura (1997) identified 
four sources of group efficacy: (1) group's past 
performance accomplishment, (2) the so-called 
“vicarious experiences” by looking at successful 
performance by someone who is similar to oneself, (3) 
verbal persuasion, that is, others' expression of faith in 



one’s capabilities and, (4) emotional and psychological 
arousal towards the task and the group. 

Furthermore, group efficacy-group performance 
relationship has been examined and a strong link was 
found in several studies (Silver and Bufiano, 1996; 
Gibson et al., 2000). While the relationship between 
group efficacy and group performance is quite robust, 
researchers do not agree about the operationalization 
and measurement of group efficacy construct. Gibson et 
al., (2000) compared three methods of measuring group 
efficacy: (1) group potency, which refers to a belief ina 
group about its general effectiveness across multiple 
tasks, (2) an aggregation of group members’ estimates, 
and (3) group discussion which is based on a consensus 
among group members about the performance of the 
group. These methods were used to assess group 
efficacy regarding several group performance 
characteristics. The authors found that discussion 
method predicted group outcomes better than the other 
two. 

Group potency is not relevant for the present research 
since there is only one task group members have to 
perform. Also, group discussion may not be appropriate 
as a single response is obtained through free format 
discussion within the group. Some members may 
influence others to reachconsensus. In the current study, 
we selected a method where members of the group 
assign an estimate of the group’s ability to perform a 
specific task without being biased by their peers. These 
estimates are then aggregated taking into account the 
variance within groups. Within-group interrater 
reliability will be calculated for group efficacy sclae 
using James, Demaree, and Wolfs (1984) interrater 
reliability index, so aggregation of data is justified. 

Team Building 

Team building refers to 

enlisting the participation of a group in planning 
and implementing change which will be mc^e 
effective than simply imposing change on the 
group from outside (Salas et aL, 1999). 

An extensive literature review was conducted on team 
building and its application to sport teams (Hardy and 
Grace, 1997). The results show that team building leads 
to an increase in team performance. On the other hand, 
Smither et al., (1996) stated that research findings on the 
effectiveness of team building provide a mix results that 
make drawing firm conclusions difficult. The reasons 
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behind these findings can be attributed to the non- 
empirical, narrative nature of most of these studies and, 
the ambiguity of what team building is. 

Indeed, team-building conceptualization dates back to 
Beer (1976) who attempted to formalize this construct 
Salas et al., (1999) built on this work and identified four 
components of team building as 1) goal setting (group 
members set objectives at the beginning of the task), 2) 
interpersonal relations (groip members develop trust in 
one another and confidence in the group), 3) problem 
solving (group members become involved in finding 
solutions to the problems encountered), and 4) role 
clarification (increased communication among group 
members regarding their roles within the group). This 
comprehensive framework for team building construct 
was adopted in this study. 

Group Performance 

Group performance has been conceptualized in terms of 
objective and subjective outcomes. Objective 
assessment includes countable measures such as group 
productivity, where as subjective measures include the 
subjective rating of the performance of the group. 
Hackman (1990) identified three dimensions of group 
performance construct. First, task effectiveness, which 
refers to the degree to which, the group outputmeets the 
standards of the organization. Second, system viability 
which refers to the degree to which the process of 
carrying out the work enhances the capability of 



members to work together interdependently in the 
future. Finally, professional growth, which refers to the 
degree to which the group experience, contributes to the 
growth and personal well being of team members. 

To date and in the best of our knowledge only one study 
(Chang and Bordia, 2001) has tested this measure of 
group performance empirically. The authors argued that 
Hackman’s (1990) measure of group performance is the 
most comprehensive one for understanding and 
capturing group performance. The importance of 
assessing this construct is critical since it is the 
dependent variable in most group-related studies. 

RESEARCH MODEL AND HYPOTHESES 

As can be seen in Table 1 (for space purpose, we show 
here only a subset of the literature), what has been done 
in prior research reflects studies that have tested links 
between one independent variable (oftentimes one 
dimension of a multidimensional construct) and a 
dependent variable (i.e., group performance). Moreover, 
clear construct conceptualization and an assessment of 
the reliability and validity of each construct, that is, the 
measurement model as well as the structural model was 
lacking in most studies (Carless and DePaola, 2000). 
Furthermore, a comprehensive research model, as shown 
in Figure. 1 , which examine the three main determinants 
of group performance, has been called for in several 
studies (Cohen and Bailey, 1997). It was the objective of 
the study presented hereto develop and test such model. 



TABLE 1 

SOME EMPIRICAL RESEARCH 



Author (s) 


Group 

Cohesion 


Group 

Efficacy 


Team 

Building 


Group 

Performance 


Gibson et al., (2000) 




X 




X 


Salas et al., (1999) 






X 


X 


Carless and DePaola (2000) 


X 






X 


Chang and Bordia (2001) 


X 






X 


Pescosolido (2001) 




X 




X 
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FIGURE 1 

RESEARCH MODEL 




Group Efficacy 

Group collective sense of capability may be an 
important determinant of its effectiveness. Group 
efficacy-group performance relationship has been a 
subject of interest and a strong link was found in several 
studies (Earley, 1994; Peterson et al., 1996; Silver and 
Bufiano, 1996; Gibson et al., 2000). Group members, 
who have a good estimate of the group’s ability to 
perform a specific task, will have higher group cohesion 
and consequently perform better. Hence, 

HI a: Group efficacy is positively related to task 

cohesion. 

Hlb: Group efficacy is positively related to social 

cohesion. 

Team Building 

Research shows that team building leads to an increase 
in team performance (Woodman and Sherwood, 1980; 
Hardy and Grace, 1997; Shandler and Egan, 1996). 
Enlisting the participation of a group in planning and 
implementing change will be more effective than simply 
imposing change on the group from outside. 
Accordingly, we hypothesized that team building leads 
to group cohesion and consequently perform bdter. 
Hence, 

H2a: Team building is positively related to task 

cohesion. 

H2b: Team building is positively related to social 

cohesion. 



Group Cohesion 

Group cohesion and its impact on group performance 
has been a subject of interest in several models of 
effective teams (Cohen and Bailey, 1997). The 
importance of this construct has been identified as a key 
in effective work groups (Evans and Dion, 1991; Mullen 
and Copper, 1994; Cohen and Bailey, 1997). 
Accordingly, we expect that IS project team members 
that show a high level of cohesiveness will have a higher 
performance. Thus, we hypothesized: 

H3: Task cohesion is positively related to group 

performance. 

H4: Social cohesion is positively related to group 

performance. 

METHOD 

Sample of Respondents 

One hundred and eighty five MIS undergraduate 
students from a 4‘^-year systems analysis and design 
course participated in this study on a voluntary basis. 
Thirty-five groups of 4 to 6 students were formed 
randomly at the beginning of the semester to conduct 
real system development projects at different companies 
in a mid-east Canadian metropolitan city. The group 
project lasted 4 months. Participants were informed that 
all data will be kept anonymous and that the research is 
intended to have a better understanding of group 
dynamics and their effect on group performance. 
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Questionnaire Items 

The constructs are measured and operationalized with 
specific items contained in the survey instrument. The 
first step in developing instruments with desirable 
psychometric properties is to specify the domain of the 
construct and be as exact as possible in defining the 
conceptual content of the construct. An examination of 
the literature was conducted; mainly group research and 
IS literature showed a deep understanding of these 
theoretical concepts. The second step, suggested by 
Churchill, of instrument development involves verifying 
that the dimensions identified in the previous step are an 
accurate representation of the constructs and generating 
items to capture the specified construct domain. 

Measures used by Chang and Bordia (2001) to assess 
task cohesion and social cohesion were each measured 
by four-item scales. Group efficacy was measured by a 
scale of one question (Pescosolido, 2001) “What is the 
highest grade level that you feel 100% sure that you are 
capable of achieving as a group?” Team building was 
measured by four item scale assessing goal setting, 
interpersonal relations, problem solving and, role 
clarification (Salas etal., 1999). Group performance was 
measured by a three subjective item scale (Hackman, 
1990). 

Analytical Procedures 

Hair et al. (1998) have underlined the importance of 
screening data prior to any analysis. The investigation of 
issues related to missing data, outliers and assumptions 
of multivariate analysis followed the authors’ 
guidelines. Screening the 185 usable questionnaires for 
missing data shows a negligible number was missing. 
Twelve questionnaires were drop from further analysis 
because they were half completed. A mean substitution 
was used as one acceptable means of generating 
replacement values for all the missing data (Hair et al. 
1998). Furthermore, outliers, which are unusual 
observations that may not be representative of the 
population understudy, were examined. Outliers can be 
the result of some data entry error or extraordinary 
events and have the potential to seriously distort 
statistical tests. The research data were analyzed to look 
for univariate and multivariate outliers. Since most of 
the variables under study were measured on a seven- 
point scale, none of the observations appear extreme and 
therefore all the data were kept for analysis. 
Furthermore, data normality was checked (PLS does not 
require data normality). These tests involve assessing 



the normality of data distributions. Data normality was 
tested using skewness and Kurtosis tests in addition to 
Kolmogorov-Smimov normality test. The results of this 
examination led us to believe that the assumption of data 
normality is accepted. 

Assessment of the Measurement Model 

Assessment of the research model was performed using 
self-report survey data and Ldhmoller’s partial least- 
squares (PLS) approach to multiple indicator structural 
equation analysis. The largest construct in this study has 
4 items, which makes a required sample by PLS of 20 
observations (4 * 5). Although it is based on ordinary 
least-squares regression, PLS is chosen over regression 
because it allows: 1) modeling multiple dependent and 
independent variables; 2) incorporates unobservable 
constructs, and 3) empirically estimates the contribution 
of multiple construct measures. In addition, PLS relaxes 
data normality constraints and it is suitable for this study 
since the research model is at the early stage of 
development. 

PLS analysis involves two steps: (1) assessment of the 
measurement model, including the reliability and 
discriminant validity of the measures, and (2) assess- 
ment of the structural model. For the assessment of the 
measurement model, individual item loadings and 
internal consistency reliabilities are examined as a test 
of reliability. For discriminant validity, items should 
load higher on their own construct than on other 
constructs in the model, and the average variance shared 
between the constructs and their measures should be 
greater than the variances shared between the constructs 
themselves. The structural model and hypotheses are 
tested by examining the path coefficients. In addition, 
the explained variance in the dqjendent constmcts is 
assessed as an indication of model fit. 

Reliability Assessment 

PLS estimates parameters for both the links between 
measures and constructs (i.e., loadings) and the links 
between different constructs (i.e., path coefficients) at 
the same time. The adequacy of the measurement model 
can be assessed by looking at: (1) individual item 
reliabilities, (2) the convergent validity of the measures 
associated with individual constructs, and (3) 
discriminant validity. Individual item reliability is 
assessed by examining the loadings of the items with 
their respective construct. A rule of thumb employed by 
many researchers (Rivard and Huff, 1988) is to accept 
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items higher than 0.707, which implies that there is 
more shared variance between the construct and its 
measure than error variance. 

The second criterion used is the Cronbach ’ s alpha which 
reflects the consistency of the measure and the 
homogeneity of the items in the scale. In the majority of 
cases, the alphas equalled or exceeded 0.7, the 
benchmarkNunnally established for instruments usedin 
exploratory research (Nunnally, 1978). 

The third indicator assessed is the Rho coefficient 
provided by PLS. This internal consistency is assessed 
in a PLS model using a measure developed by Fornell 
and Larcker (1981). It is not influenced by the number 
of items in the scale as opposed to alpha; it is; however, 
influenced by the relative loadings of the items. It is 
based on the ratio of construct variance to the sum of 
construct and error variance. A value greater than .50 
indicates that the construct variance is accounts for at 
least 50 % of the measurement variance. 

Finally, a measure called the Average Variance 
Extracted (AVE) determines the average variance shared 
between a construct and its measures and the variance 
shared between the constructs, which are the squared 
correlations between the constructs. AVE is obtained by 
the sum of loadings squared, divided by the number of 
items in the construct, whereas the variance shared 
between two constructs corresponds to the square of the 
coefficient of correlation between the latter. The AVE 
should be higher than 50 % (Rivard et al., 1993). 



TABLE 2 

RELIABILITY ASSESSMENT 



Variables 


#of 

items 


Alpha 


Rho 


AVE 


Team Building 


4 


0.74 


0.88 


0.66 


Self-Efficacy 


1 


— 


— 


— 


Task Cohesion 


4 


0.90 


0.90 


0.78 


Social Cohesion 


4 


0.84 


0.87 


0.62 


Group 

Performance 


3 


0.81 


0.95 


0.86 



Convergent and Discriminant Validity Assessment 

Convergent validity can be assessed by the degree of 
“agreement” among items measuring the construct. To 
evaluate discriminant validity, Fornell and Larcker 
(1981) suggested a comparison between the average 
extracted variance of each factor and the variance shared 
between the constructs (the squared correlations 
between the constructs). 

A PLS run was conducted to compute the covariance 
matrices of all measures used o evaluate the loadings of 
the different measures on their constructs. Table 3 
reflects the loadings of items on their own constructs. It 
is expected that the loadings of all variables within the 
same construct should be high on this construct, 
indicating high convergent validity, and low on the other 
ones, displaying high discriminant validity. The first 



TABLE 3 



PLS LOADINGS: CONVERGENT AND DISCRIMINANT VALIDITY 



Loadings 


Variables 


Task Cohesion 


Social Cohesion Self-Efficacy 


Team Building 


Group Performance 


TSKl 


0.85 








TSK2 


0.91 








TSK3 


0.87 








TSK4 


0.71 








SOCl 




0.78 






SOC2 




0.80 






SOC3 




0.85 






SOC4 




0.71 






EFF 




1.00 






TMl 






0.84 




TM2 






0.86 




TM3 






0.75 




TM4 






0.79 




PRFl 








0.90 


PRF2 








0.94 


PRF3 








0.94 
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characteristic indicates that they share a lot of variance 
with their construct and the latter that they are 
independent from the other constructs. Table 4 reflects 
AVE (average variance extracted) values. Squared 
correlations (off-diagonal) are reported where the on- 
diagonal are the AVE squared root. The number on the 
diagonal should be higher than the elements off 
diagonal. 

These loadings show a clear discriminant and 
convergent validity for all constructs. Items that did 
show an imperfect loading structure were deleted, that 
is, they did not show (1) higher loadings on their 
respective variables displaying convergent validity and 
(2) lower loadings on the other variables displaying 
discriminant validity. Items of the two dimensions of 
Asset Specificity construct (supplier investments and 
human resources) were aggregated by averaging, for 
each dimension, the weight of each item and its 
associated value. Then, PLS run was performed and 
items that did not load high on their constructs were 
deleted. The reasons for these findings are not entirely 
clear. One possible explanation is that the validity of 
this measure cannot be truly established on the basis of 
a single study. Validation of measures is an ongoing 
process, which requires the assessment of measurement 
properties over a variety of studies in similar and 
different contexts. 

TABLE 4 

VARIANCE SHARED BETWEEN CONSTRUCTS 



V ariables V ariance 



Taks 

Cohesion 

Social 


0.88 








Cohesion 

Team 


0.07 


0.79 






Building 


0.37 


0.46 


0.81 




Group 

Performance 


0.62 


0.03 


0.24 


0.93 


Self-Efficacy 


0.28 


0.08 


0.20 


0.27 1.00 



The numbers on-diagonal are the square root of the 
average variance extracted (AVE). These values should 
be higher than that off-diagonal. The largest correlation 
off-diagonal is 0.62. The lowest AVE squared root (on- 
diagonal) is 0.93. Hence, the smallest on-diagonal value 
is larger than largest off-diagonal values displaying the 
expected pattern. Hence, from the results of this second 
PLS runs, discriminant and convergent validity of the 



measures appeared to be satisfactory. The measures can 
be used to further assess the model and verify the 
hypotheses. 

Assessment of the Structural Model 

The structural model shown in Figure 2 provides the 
hypothesized relationships. The model’s constructs are 
conceptualized with reflective items since these items 
reflect their respective construct by covarying 
simultaneously. Each hypothesis was tested using PLS 
Graph (Chin, 1995) by looking at the path coefficients. 
The estimated path effects are given along with their 
degree of significance. A bootstrapping procedure was 
used to assess the level of significance of the paths 
computed by PLS. T-values were computed from a 
series of PLS evaluations made against several partitions 
of the data set. The results of the PLS run with the 
overall sample (n = 35 groups) are shown in Figure 2. 

FIGURE 2 

MODEL PARAMETERS 
FOR THE RESEARCH MODEL 




•; p<0.05; p<0.01 



As can be seen in Figure 2, the correlations between the 
constructs suggested that there were a priori grounds for 
expecting significant effects between them except the 
ones between group efficacy and social cohesion and the 
correlation between social cohesion and group 
performance. 

Self-efficacy shows a positive effect on task cohesion 
with a path coefficient of 0.31 (p < 0.05) but not on 
social cohesion. Team building shows, as expected, a 
positive effect on task cohesion with a path coefficient 
of 0.47 (p < 0.05) and a significant effect on social 
cohesion with a path coefficient of 0.69 (p < 0.01). Task 
cohesion shows a strong effect on group performance 
with a path coefficient of 0.79 (p < 0.01). Surprisingly 
social cohesion had no effect on group performance. The 
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percentage of the variance explained (R^) of group 
performance was 62 %. 

DISCUSSION AND CONCLUSIONS 

The primary objective of this research was to examine 
the antecedents and consequences of group cohesion, 
that is, task cohesion and social cohesion. To that end, 
the developed model was empirically tested using data 
collected during a survey study. The results of the 
empirical analysis provide a number of interesting 
insights and suggest several conclusions. 

The findings of this study provide support for the group 
research theory perspective on the determinants of group 
performance. First, group efficacy was found to play an 
important role as an antecedent of task cohesion but not 
social cohesion. Task cohesion is increased when group 
members collectively believe that they will perform 
better on specific task. Group members were united in 
trying to reach their goals as a groip and were 
committed to the task. However, group efficacy did not 
have much effect on social cohesion. One possible 
interpretation of this result is that even though group 
members believed that they were capable of achieving 
highest performance, they did not develop social ties 
outside of group work. These are very interesting 
findings. Groups with high degree of self-efficacy will 
develop cohesion around the task in hand and not a 
social one. 

As advocated by prior research, the analysis also sheds 
light on the positive effect of team building cn both task 
and social cohesion. As expected, there was a significant 
positive effect. Group members who set objectives to 
achieve the group’s goals, share information regarding 
their respective roles within the group, and were 
involved in the identification of major problems in the 
group and implement solutions for those problems 
developed both task and social cohesion. The argument 
posits that the ever-p*esent team building ingredients 
will induce group members to a higher degree of 
cohesiveness. 

Finally, and in accordance with group research 
literature, there was a significant positive impact of task 
cohesion on group performance but no effect of social 
cohesion on group performance. These results may 
imply that in the presence of higher degree of task 
cohesion, group will have higher performance. On the 
other hand, groups that may spend time partying 



together and socializing did not show any significant 
correlation with group performance. Indeed, these 
results provide insightful information as of the 
predictors of group performance and the role of group 
efficacy and team building on task cohesion. 

IMPLICATIONS FOR 
RESEARCH AND PRACTICE 

There are several potentially important implications for 
IS researchers and practitioners. The findings of the 
study demonstrate the value of the contributions of the 
antecedents and consequences of group cohesion, more 
particularly, task cohesion. This study represents an 
initial step in a theory-based empirical test of group 
performance and its antecedents. In developing and 
testing the research model and lays the foundation for 
future research concerning this important issue of group 
performance IS investigators interested in this issue will 
have instruments to capture its determinants. 

For practitioners, this study has some important 
implications. The results of the survey received a 
substantial support for the theoretical foundation upon 
which this research is built on. The interplay of the 
relationships of different constructs is also of interest to 
practitioners who assess group performance. Certainly, 
it is hoped that firms will be able to enhance their 
understanding of what makes group perform better by 
employing the instrument. 
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ABSTRACT 

A significant number of businesses world-wide have implemented enterprise resource planning systems in order to 
achieve improvements in transaction processing efficiency, process integration and productivity and lead a 
significant paradigm shft in managing business operations. However, with too much focus on specialism, traditional 
curricula of business education did not make any provision for the holistic approach integrating the different aspects 
of business operations. Recognizing the importance of process centering, enterprise-wide computing solutions 
available in the market, and increasing role of information in managing business operations, apostgraduate business 
course in enterprise resource planning using SAP R/3 system as a teaching and learning aid at the University of 
Western Sydney, Australia was designed and implemented in 1998. This paper briefly reviews the program after four 
years of program delivery, and expounds the background to this program, its unique design features, delivery 
processes, and assessment systems from the educational effectiveness perspective. This paper reports on the 
work-in-progress of an evaluative research study of the innovative program that incorporates SAP R/3 system. Based 
on the feedback from students and preliminary analysis of the data collected the program appeared to have 
succeeded in transforming the students into business specialists with strong process, software and project 
management skills. However, complexity of the software, limitations of academic skills and knowledge, lack of quality 
curriculum materials, reliability of the system, support facilities, varying student mix, difficulty in obtaining the 
industry support, and effectiveness of the curriculum are observed to be problematic. 



INTRODUCTION 

Taking into consideration the latest developments in 
enterprise wide computing and increasing role of 
information in managing operations, and the need to 
design appropriate professional courses to meet the 
requirements of industry, a post graduate business 
program in enterprise resource planning was designed 
and implemented at the University of Western Sydney, 
Australia. This paper describes the background of the 
innovative design of this masters program and reports on 
the evaluation of the program after four years of 
running. Based on the information collected through a 
survey of students and lecturers, and the principles of 
curriculum design, delivery and evaluation in higher 



education context, this paper evaluates the program and 
recommends changes for future. 

BACKGROUND AND CONTEXT 

The environment in which business operates today is 
changing very rapidly. E-business initiatives, increasing 
competition, cross-functional integration, integration of 
inter-enterprise processes in the value chain, increasing 
strategic importance of information technology are 
forcing business organizations to change their 
management systems and processes rapidly and 
frequently (Handfield& Nichols 1999) in line with the 
dynamic changes in sipply, demand and customer 
expectations. Role of information and computer-based 
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information system in business and their significance is 
increasing as new technologies and new processes are 
evolving. From traditional payroll and bookkeeping 
functions, computerized systems are now penetrating 
complex managerial areas. 

Move Towards Integration and Processes 

Many organisations such as General Electric, Toyota, 
IBM, Ford, Microsoft and others are shifting the focus 
of management development away from specialization 
and towards integration of the different disciplines and 
functional departments (Malekzadeh 1998). This way 
they believe, will be able to provide increasingly 
demanding customers with a better and more flexible 
service. In this context, business organisations are using 
information technologies as enabler of change, and 
integrated enterprise-wide computing information 
systems to support today’s complex and effective 
decision making (Davenport 1998). Enterprise systems 
are integrated systems that provide seamless integration 
of all the information flowing through a business 
organisation from accounting, human resources, supply 
chain and customer information. Usage of ERP software 
is changing the face of information technology in 
business organisations and is blurring the lines between 
information technology and users. Increasingly, the 
managers are required to become information 
intelligent, with the ability to learn new information 
technology applications and design and redesign 
systems. 

Business Education in Universities 

As the organizations adapt in response to changing 
environmental conditions, there should be a 
corresponding change and adaptation in universities. 
The strength of business education at present lies in 
specialist or technical areas such as marketing, finance, 
accounting, human resource management, or operations. 
All these fields evolved over a period of time into 
narrow areas of study. These functionally specialized 
schools within university structures reflect the needs of 
the large, highly bureaucratized organizations that have 
dominated this century. Not surprisingly, universities 
are generally criticized of producing narrowly trained 
individuals who hold a compartmentalized orientation of 
the way business is conducted (Morris 1997). 

Reports after reports revealed that business graduates do 
not have the ability to integrate and cross-functiaial/ 
interdisciplinary perspective (AC Nielsen 1998, Barker 



et al. 1998, Karpin 1995). For example, Karpin pointed 
out that the universities in Australia though successful 
in providing functional skills, have precluded the 
development of integrative skills critical in business 
environment. The report argues that over-emphasis on 
functional skills such as accounting andmarketing act as 
a barrier to building better teamwork skills and 
integrated approach to business management (Karpin 
1995). Though adopting cross-functional integration as 
a teaching approach and creating an information 
orientation were recommended in a classic review of 
higher education by Porter and McKibbin in 1988, the 
evidence of such change is sketchy. 

The integrative business processes that span across 
various functions are not embedded in the present 
curriculum designs and generally left entirely to either 
the individual student or to a learned faculty member to 
tie them together basing on his/her experience. Though 
debates are on about an alternate business education 
curriculum integrating the common core with a process 
oriented approach, the forums for discussion and 
exchange of ideas concerning these redesign efforts are 
limited (Stover et al. 1997). The exponential growth in 
business management knowledge and the increasing 
technical content of even the most elemental business 
course has not given chance for the additional 
integrative elements of the course. 

Graduates and employers are realizing the importance of 
integrative capability at the workplace and are forced to 
spend considerable amount of resources in retraining 
business graduates and impart integrative 
process-oriented skills. Thus it triggers an unlearning 
phenomenon within the fresh graduate to become 
acceptable employee and start learning. This learning 
process becomes harder and harder as the graduate is not 
equipped with the skills to learn, reflect and learn 
continuously on his/her own. Thus the educational 
system that was designed based on the old management 
paradigms relevant to a manufacturing-based economy, 
and in a time of industrial revolution, has now become 
irrelevant and obsolete. 

Potential Role of ERP Systems in Business 
Management Education 

In universities, too much of focus to integrate IT into the 
business curriculum has been on either the acquisition of 
IT skills or usage of IT to support the teaching process 
(Mutch 1996). While accepting that these are necessary 
developments, more attention is needed in imparting the 



238 



Proceedings of the 17^ Annual Conference of the International Academy for Information Management 



meaning of information and its critical role in operations 
management. Enterprise resource planning (ERP) 
systems provide an excellent opportunity for introducing 
business process orientation and help in developing a 
cross-functional curriculum that imparts integrative 
skills and offer real-world exposure to students (Watson 
& Schneider 1999). By integrating enterprise resource 
planning systems into the curriculum, it is possible to 
overcome some of the problems associated with the lack 
of integration and process-orientation in business 
management education, to some extent. 

For example, its process-oriented approach to the 
business, seamless integration of data that interfaces 
several functions, ability to use mathematical models, 
focus on information and systems view of decision 
making, and the ability to demonstrate the affect of 
integrated data in business will highlight the challenges 
ahead and prepares them better than an education that is 
focused on functional specialisation. Importantly ERP 
system’s ability as a powerful computing tool to 
demonstrate the cross-functional integration and the 
influence of business process approach and 
reengineering in managing operations play a powerful 
role in preparing managers for the century. 

RATIONALE AND APPROACH TO 
CURRICULUM DESIGN 

There are different approaches of incorporating ERP 
systems into the business curriculum and depend upon 
the field or discipline initiating this move (whether 
information systems, computer science or business 
management), and the specific focus (particular software 
training or business process perspectives or as an aid to 
teaching business/information concepts). The employers 
embracing enterprise systems are dependent upon 
vendor-specific product specialists who are information 
technology specialists and lack theoretical foundations 
in business management and process-centered approach 
to managing business (Elam et al. 1999). In the absence 
of process focused specialists, a functional specialist 
generally handles the reengineering and configuration of 
the processes within functional framework and tries to 
integrate them with other functions. Similarly, 
information technology specialists translate the 
‘reengineered’ processes by the functional specialists 
into an automation framework. This approach may result 
in gap between the process and technology and may lack 
the balance between business and technology. The 
enterprise solutions thus developed may become 



complicated, time-consuming and expensive exercise for 
the business (Bingi et al. 1999; and Booth et al. 2001). 
In addition, absence ofprocess-centered approach inthe 
implementation negates the very objective of using 
enterprise-wide resource planning solutions. 

Balance Between Scholarship and Vocational 
Training — A Dichotomy 

Integrating a large ERP software product into business 
course raises a traditional dichotomy in an ideal 
university curriculum. Resolving the question of balance 
between scholarship and vocational training is 
imperative to meet the changing market needs of 
universities and their customers — industry and students. 
Contrary to the traditional ‘push’ model in designing 
and developing courses, a realistic ‘market pull’ models 
are increasingly becoming popular in Australia. These 
new and revised curriculumsare, through most subjects, 
increasingly, are reflecting a vocational enphasis 
embedded within a suitable theoretical or academic 
framework. 

Despite its prevalence, demand for graduates and 
success of formal academic business and management 
education, serious questions have been asked about their 
relevance to a modem industrial economy. Some of 
these educationalists, who prefer a classical approach to 
university education, doubt the value of application or 
vocation in business education (Cannon 1996). More 
subjects with a practical orientation would be 
recognized as a valid pathways to further and higher 
education, thus elevating the status of the more 
vocationally oriented subjects. In fact, a report by 
Cunningham et al. (1998) recommended universities to 
develop strategies to capture the profitable professional 
graduate market, which is vocational to some extent, as 
a component of life long learning. The educational 
needs of working professionals, expanding international 
student markets, and budgetary pressures on universities 
to earn fees, have encouraged several universities to 
start full-fee paying post graduate programs that will 
impart employable skills to the students. Several 
specialized business courses were designed and 
successfully marketed by the universities . Some of them 
are in quality management, marketing management, 
logistics management, operations management, human 
resource management, industrial relations, occupational 
health and safety, finance and banking, professional 
accounting, public sector management, and technology 
management. 
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In line with the growth of enterprise resource planning 
software solutions and related information technologies, 
demand for highly skilled professionals is rapidly rising. 
The software manufacturers have their own training and 
certification programs to deliver product-specific 
application, configuration and implementation skills to 
enterprise IT professionals and consultants. Working in 
conjunction with operational staff in the customer 
organization, the “vendor-certified” software product 
experts undertake the complex task of implementing 
enterprise solutions in accordance with the respective 
vendors’ recommended practices. 

By implementing enterprise systems, organisations are 
embarking on a new way of life that involves continuous 
changes. Changes such as adaptation of system to 
changes in the structure, installation of new releases of 
software, training new users of the system, training old 
users on new functions and capabilities and changing the 
system or software from time to time based on the 
managerial advice, will continue forever after 
implementing enterprise systems (Davenport 2000). 
Organisations must be prepared to change on an ongoing 
basis and employ resources and strategies to manage 
those changes. Though there are a large number of 
consultants who have acquired the product specific 
technical skills, they generally do not possess adequate 
theoretical grounding in business operations required for 
the day-to-day operations and maintenance. The high 
cost of these consultants and their narrow focus on one 
or two software modules in an ERP system, aeate 
problems for ongoing running and maintenance of the 
enterprise systems after their implementation. This 
program is expected to fill this gap to some extent, by 
providing graduates with business knowledge from 
process centred perspective as well as software skills. 
This initiative is expected to support larger enterprises 
to manage change in a cost-effective way, and a large 
number of medium and smaller organizations to 
affordably implement and maintain integrated ERP 
solutions in their operations. 

SAP and other ERP Software Solutions — The Choice 

Several enterprise resource planning software products 
such as SAP R/3, BAAN, PeopleSoft, and JD Edwards 
have emerged in the marketplace. These software 
products have a complex client-server architecture, are 
powerful and user friendly, web-enabled and 
incorporates several decision support capabilities now, 
and cover abroad spectrum of enterprise functions. In 
the 1990s, as a part of their strategic moves of 



expanding the adoption of ERP software, several 
software vendors developed university alliance 
programs and started encouraging universities to 
incorporate the software into their curriculum. SAP, 
People soft and JD Edwards are leadings in this 
initiative. Though started in Europe initially, and then to 
USA, these initiatives moved into the Asia and 
Australasian region in mid 1990s. In its university 
alliance program, SAP provides universities with a 
complete version of its R/3 system software and a 
training database, along with associated documentation, 
technical support and faculty training free of cost 
(Watson & Schneider 1999). Universities on their part 
will invest in the hardware and integrate SAP R/3 in 
their curriculum. 

Because SAP was the most popular ERP software with 
about 40% market share in 1996, and the prohibitive 
cost of other competitor products from JD Edwards and 
PeopleSoft, and lack of a viable university support 
programs at that time from other competitors, SAP was 
chosen by the University of Western Sydney in the year 
1997. When the author made first contact with the SAP 
Australia in 1996, only two universities, the Queensland 
University of Technology (QUT), Brisbane, and, the 
Victorian University of Technology (VUT), Melbourne 
had just started some courses on ABAP programming 
and SAP. These courses were offered from the Faculties 
of information technology and from the schools of 
information systems. Considering the limited 
investments required for hardware and free availability 
of software from SAP and not from any other ERP 
vendor, the University of Western Sydney Nepean, has 
started designing the curriculum incorporating SAP R/3 
in their business operations courses in its then successful 
masters program in operations management. 

A new professional master of business course in 
enterprise resource planning was designed to develop 
lateral depth to students across all elements of business 
operations. Some of the objectives associated with an 
ERP system initiative in universities include exposing 
students to the real-world from the classroom, 
developing cross-functional curriculum, enriching 
specific curriculum, exploring new research 
opportunities and creating a competitive advantage 
(Watson 1998). In addition, incorporating ERP software 
into curriculum is likely to enhance the credibility of 
business schools in the eyes of industry that generally 
criticizes academia of ignorance of the developments in 
the industry. 
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The main objective of this program was to equip 
students with the knowledge of the business processes, 
and skills for redesign, configuration and 
implementation of these business processes with the 
help of SAP R/3 software. In order to fill the skill gap 
and to deliver broad-based expertise in emerging ERP 
solutions discipline, the proposed academic program 
envisages in its curriculum the following. It includes 
business operations theory from process management 
perspective, a holistic view to business management, 
focus on strategic as well as day-to-day operational 
issues, a complete understanding of the enterprise 
environment, ability to redesign processes employing 
basic reengineering principles, ability to implement and 
use an ERP software in a business organization, 
awareness of the technology behind the software and 
skills to manage change. 

In 1997, no academic program in Australian universities 
provides knowledge and expertise to students in 
maintaining and assisting the ERP system 
implementation, and in providing support to users, from 
business perspective with an underpinning theoretical 
framework. There were, however, some successful 
programs in information systems area incorporating 
ABAP (programming language), process modeling and 
other business perspectives by the computer science 
faculties/schools using SAP R/3 system as a base 
(Watson 1999; Hawking 1999). Though ERP concq)ts 
can be taught without having access to a SAP R/3 
software system or any other software, hands-on 
exposure for students strengthens students’ learning 
experience and provides real-world exposure. 
Importantly, students will be able to develop ERP 
software skills that are highly valued by business. 

FEATURES OF THE PROGRAM 

This program aims to bridge the gap between traditional 
business and information technology courses offered by 
universities and the real world requirements of business 
in practical application of skills. The objective of this 
program was to equip students with necessary skills in 
optimizing and planning business processes and the 
usage of software, with knowledge of technology, 
reengineering and management of change issues 
underpinning the course. Graduates of this course are 
expected to be able to assist in implementing and 
configuring the ERP software to suit specific 
requirements of business and assist in the reengineering 
of business processes for improvement. After the 
implementation of the enterprise systems, these 



graduates are expected to maintain and upgrade the 
system. These graduates are not expected to be technical 
(information technology) personnel who have the 
capability to design and administer the system; their 
expertise, rather, will be confined to the implementation, 
configuration, running and maintenance of the system in 
several application areas. Each subject included in the 
course will not only teach the relevant conceptual 
frameworks and tools, but also develop abilities to 
extend knowledge through research and independent 
investigation. 

The Masters degree requires completion of 12 subjects. 
Graduate Diploma requires 8 subjects and the Graduate 
Certificate requires 4 subjects. The Graduate Diploma 
and Graduate Certificate are nested within the Masters 
degree. The proposed program structure has three tiers. 
The design process adapted an ‘interactive model’ of 
course design, which involves specification of learning 
outcomes, sequencing of content, selection of learning 
strategies and methods, and selection of assessment 
procedures that are common in any design processes 
(Cunningham 1997). The program structure envisages 
three tiers of knowledge with some flexibility for 
students to acquire additional theoretical inputs from IT 
or business subjects. The first level provides foundation 
knowledge, the second tier imparts knowledge and 
practical skills about business processes and 
management systems, and the third tier focuses on 
strategic technology, enterprise performance 
management and implementation issues. 

Foundations 

In the first tier of the program structure, three subjects 
that provide foundation knowledge to the course are 
included. They cover issues such as basic IT concepts, 
enterprise computing concepts, process concepts and 
logistics/supply chain operations knowledge. 

1 . Enterprise computing concepts — enterprise-wide 
computing concepts, enterprise application 
integration concepts, integration of processes and 
information systems, client/server technology and 
security, communications, messaging and workflow 
concepts. 

2. Commercial IT Environments — to provide basic 
concepts of systems analysis & design, database 
concepts, and data communications & network 
concepts. 
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3. Logistics and supply chain management or 
operations management — overview of logistics and 
supply chain management concepts, processes and 
costs, transportation, inventory, production, 
warehousing, supply chain integration, and 
information technologies in supply chain 
management. 

Process Management 

In the second tier, subjects related to the various 
business processes are included. They include both 
primary business processes and enabling processes. This 
level provides knowledge and skills concurrently on 
several process-focused modules emphasizing on the 
interface with other modules/processes. 

1. Customer order management processes and 
systems — includes concepts of customer order 
management and customer service; incorporates 
Sales & distribution and Customer service modules 
of ERP software. 

2. Production Management processes and systems — 
imparts concepts and basic knowledge of production 
planning and control processes and systems for 
repetitive or batch manufacturing system; and 
incorporates production planning/management 
modules of ERP software. 

3. Materials and quality management processes and 
systems — imparts knowledge of procurement 
processes and incorporates MM module of the ERP 
software. 

4. Financial planning and control processes and 
systems — imparts basic financial planning and 
control process concepts and knowledge; and 
incorporates ERP software skills in Financials and 
costing modules. 

5. Human resource management processes and 
systems — imparts basic knowledge of human 
resource management processes, and incorporates 
human resource module of the ERP software. 

Strategic and Management Issues 

Third tier of the structure incorporates strategic and 
management issues in relation to the technology and 
system management, programming languages, enterprise 



performance measurement and management, and an 
action learning project. This action learning project is an 
industry-based project that involves redesigning process 
or processes, and/or implementing parts of ERP systems 
in business organizations. Integration of business 
technology with the reengineered business processes at 
the enterprise level forms the core in this tier. 

1. Managing Integrated Business Technology — 
incorporates ABAP programming, reporting and 
analysis, and system administration issues in ERP 
context. 

2. Enterprise Reengineering and Management of 
change — Reengineering of processes and 
implementation of SAP with the help of ASAP 
methodology, and general concepts of change 
management and project management; and 
integration of the technology and various other 
subjects in the course that emphasize on particular 
aspects or processes. 

3. Enterprise Resource planning project — action 
learning project involving application of concepts, 
models and skills to the actual live project in 
industry; with cooperation and involvement of 
industry partners of the program. 

4. Value Based management systems — new 
cutting edge developments such as SCM (Supply 
chain management), CRM (Customer relationship 
management), PLM (Product Life cycle 
management), SEM (Strategic enterprise 
performance management) are expected to be taught 
in this subject. In addition, students are exposed to 
the latest cutting edge software products in this area 
and their integration with ERP software. 

Individual Choices 

Students are given an option of picking up two electives 
from any school within the University. The curriculum 
designed in this program is flexible and can meet the 
needs of a broad spectrum of professionals entering this 
field. While the core requirements expose students to the 
foundation, process management, and strategic issues in 
designing and implementing ERP solutions, flexibility 
in electives allow students to tailor the program to meet 
individual career goals, depending upon their 
background and experience or lack of it. 



242 



Proceedings of the 17^ Annual Conference of the International Academy for Information Management 



Entry Requirements 

The proposed program is open for people from any 
undergraduate discipline. Applicants with an extensive 
industry experience, but no undergraduate degree of 
graduate diploma are also eligible to apply. Applicants 
are required to demonstrate their aptitude and interest in 
the field of study though a written submission and a 
letter of support from their employer or some academic 
references. This multidisciplinary entry to the program 
strongly reflects the process-oriented approach in any 
business and the overarching presence of business 
operations in any area of work. The multidisciplinary 
nature of entry is also expected to facilitate 
cross-fertilization between various disciplines and 
exchange of information and knowledge between 
students, and better understanding of process approach 
to business management, efficiently facilitated by 
academics. 

Teaching and Learning Strategies 

Delivery mode is an essential part of a teaching and 
learning. Literature cites that the mode of instruction 
should be selected according to the age and educational 
background of the participants, preferred learning styles 
of participants, available resources for course 
development and delivery, and whether the course is 
predominantly skill-oriented or knowledge based 
(Donovan et al. 1999; Holland 1995; Cunningham 
1998). Though there are three modes, viz., classroom 
based, self-paced, and computer based, no exclusive 
strategy is effective. Hence, a mixed strategy was 
proposed for the delivery of the program. In addition to 
the classroom lectures interspersed with case studies, 
practical work and assignments using software, tutorials 
and guest lectures from industry were proposed. 

Assessment is an integral part of learning. In many 
academic institutions, assessment is viewed as a way to 
measure the learning that has taken place during the 
semester, as a way to apportion grades that differentiates 
between students, but not as a way that enables students 
to demonstrate their understanding. Despite the 
necessity to award grades at a university, the assessment 
in this program is approached primarily as a means of 
giving feedback to students on their learning and getting 
feedback from students on teaching. Considering this 
objective, appropriate assessment methods were 
selected, designed, and implemented considering the 
principles of validity, reliability, feasibility and 
flexibility. In addition to these, transparency of 



processes and criteria, simplicity in procedures and 
tasks, and maintaining relevance to the changing subject 
content are the other important principles that are taken 
into consideration in the assessment process. As a 
general guideline, each subject will have three to four 
assessment tasks that comprises of examinations, 
computer-based practical examinations and tutorials, 
assignments that require theoretical knowledge and 
practical software skills, research p^ers on specific 
topics, and field project stressing on application and 
implementation at a workplace. 

EVALUATION OF THE PROGRAM— 

AN APPROACH 

Evaluation in Higher Education Context 

Evaluation in the higher education context, is the 
process of delineating, obtaining and providing 
descriptive and judgmental information about the worth 
and merit of a program’s goals, design, implementation 
and impacts in order to improve the program and 
promote understanding of the process (NRC 1 999). This 
is carried out with a view to improve the course for 
future cohorts of students. Depending upon the reason 
for doing evaluation, the type of data required and the 
sources from which data is collected is different. Some 
of the sources are the subject outline documents, 
information from subject coordinators, former and 
current students, and professional evaluators. This 
information about the course is gathered from the 
students with the help of structured questionnaires 
and/or interviews that seek information about their 
reactions to curriculum design, course content, course 
delivery, and their use of course materials, suitability of 
teaching methods employed, suitability of assessment 
procedures and methods used, and other supporting 
facilities for learning. Methods for gathering data 
include subjective judgments, student achievements and 
surveys of student opinion. 

Most evaluations in several universities depend heavily 
on information obtained from students currently 
enrolled, for evaluating the courses and improving 
teaching programs. While emphasising that there is no 
single fixed approach to evaluation, literature provides 
several approaches to evaluation — goal oriented 
evaluation, decision oriented evaluation, goal free 
evaluation, illuminative evaluation, and transactional 
evaluation (Wagner, 1994). As summarised by Wagner 
(1994), Parker’s approach to evaluation focuses on the 
performance of the participant on the job, his/her 
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satisfaction, and the knowledge gained by them, whereas 
Bell system emphasises on several outcomes in terms of 
the participants’ reactions, the knowledge gained or 
demonstrated competence, and the value of training in 
relation to its cost. Similarly, the GIRO approach deals 
with the contextual issues in evaluation and participants’ 
reactions for essentially improving the program. 
Kirkpatrick approach concentrates on the reactions of 
participants, the measurementof learning, behaviour and 
the results of training in terms of several organisational 
performance indicators. Each approach has its 
advantages and limitations, and hence the lecturer could 
use a combination of approaches to suit his or her 
specific needs and focus in the evaluation. 

Objectives of This ERP Program Evaluation 

The main objectives ofthis evaluation are to bring about 
improvements in the curriculum and delivery and to 
judge the success or failure of this innovative post 
graduate business course in enterprise resource planning 
with SAP R/3 system. With these objectives, it is 
decided to evaluate the following aspects of the 
course — curriculum design, delivery and teaching 
performance, assessment, university resources, SAP R/3 
system facility, projects, placement and other support 
facilities. There are no separate evaluations of particular 
subjects, instead, graduate students are asked to evaluate 
the entire program and provide feedback. 

Sampling and Data Collection 

Unlike other evaluations, the study focused on the 
graduate students who had completed the course and left 
the university. For the sake of definition, graduate 
student is defined as one who had completed at least 
Graduate Certificate program and left the university at 
the time of survey. The cohort of students comprises of 
international students from countries such as India, 
Pakistan, Thailand, Indonesia, Taiwan, China, Hong 
Kong, Sri Lanka, Jordon and Scandinavian countries. 
Statistics indicate that about 1 04 students completed this 
course with graduate certificate or graduate diploma or 
a Masters degree, in the past four years. The {xogram is 
offered mostly in the evening and on weekends to 
accommodate for the part-time students who comprise 
about 50% of the students in the course. 

The type of instruments employed for data collection in 
this evaluation is questionnaires. Though literature 
suggests different other methods such as case studies, 
supervisor ratings, logs, and interviews, a questionnaire 



survey is employed to collect data. This approach is 
considered to be appropriate and relevant here, in view 
of its low cost, confidmtiality and relatively easier way 
to administer and understand, and avoidance of fear and 
embarrassment for students (Bums, 1999). Since the 
questionnaire is administered on all the students, there 
are no sampling problems. However, there are certain 
limitations for this method — viz., possibility of 
misrepresentation of the questions by the respondents, 
incomplete and inaccurate information, ambiguity or 
vagueness in the questions designed, non-flexibility in 
providing free expression of opinions and finally the 
motivation of the respondents to answer questions (Gay 
and Diehl, 1992). However, whatever form, the 
importance of student feedback is immense in making 
improvements in the program design and delivery, 
particularly in the university’s push towards more 
customer focused educational programs. 

ANALYSIS AND TENTATIVE FINDINGS 
Demographics 

An analysis of the demographic data reveals that about 
50% of the students are international students coming 
from various countries such as China, India, Indonesia, 
Jordan, Norway, Pakistan, Sri Lanka, Taiwan, and 
Thailand. While all the local students have some 
experience in either information technology or business 
functions or in both, only about 50% of international 
students have some experience before enrolling in this 
course. Summary of the respondents’ demographics are 
given in Table 1. 

Overall Perception of the Course 

On a scale of 1 to 5 (1= very poor, 2 = poor; 3= good, 
4= very good and 5=excellent), the respondents are 
asked to assess the course with reference to various 
dimensions, as well as the knowledge and skills gained 
by them in various areas. Analysis reveals that students 
on the whole, have perceived the curriculum design as 
good, while other dimensions such as resources and 
support, teaching performance are considered 
inadequate. The overall evaluation of the program with 
reference to various dimensions is given in Table 1. 

Differences Between Local and International Students 

An analysis of their responses and t-tests revealed that 
there are significant differences between the 
international students and local students. This 
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TABLE 1 

SUMMARY OF THE RESPONDENTS’ DEMOGRAPHICS 



Characteristics of respondents 


Local students 


International 

students 


Overall 


Proportion of respondents 


48% (50) 


52% (54) 


100% 


Response rate 


50% (25) 


30% (16) 


39% 


Students with ‘some experience’ 


100% (25) 


38% (6) 


76% 


Students with ‘No experience’ 


0% (0) 


62% (10) 


24% 


Students ‘With IT experience’ 


68% (17) 


19% (3) 


49% 


Students ‘With Functional experience’ 


60% (15) 


19% (3) 


44% 


Students with both ‘IT and Functional experience’ 


28% (7) 


0% 


17% 


Students with undergraduate degree 


74% (20) 


100% (16) 


88% 


Students with exposure to ERP software 


24% (6) 


7% (2) 


20% 


Students working in an organization that has implemented or 
intend to implement SAP software 


36% (9) 


0% (0) 


22% 



classification is made without any prejudice, and 
keeping in view of the equity issues. The purpose here 
is to delineate the differences in terms of learners’ needs 
and perceptions - both international and local and try to 
improve the program to consider both aspects. Results 
show that international students are relatively less 
satisfied with the course than the local students. The 
differences are particularly significant with reference to 
curriculum design, course delivery and the overall 
program. A review of the comments and the informal 
discussions revealed that the international students are 
more concerned about the curriculum design and 
resources & support issues while local students are 
concerned about the knowledge and skills of academic 
staff and course materials. A summary of the differences 
are presented in Table 2. 

Further analysis, however, reveals that the satisfaction 
level of graduates is significantly different for different 
aspects. From the above, it can be seen that the 
satisfaction level of students in terms of resources and 
support and the course delivery are not good enough, 
when compared to the curriculum design. However, the 
comments made by the students and rating of other 
factors reveal some of the weaknesses in the curriculum 
design and delivery as well as the resources and support 
provided to the students. A brief review of the issues 
raised by the students is given below. 

Curriculum Design and Sequencing of Subjects 

Curriculum design is considered the strength of this 
course. Confirming this notion, respondents rated it to 
be good and reinforced the university’s belief about the 
program. Certain elements of the design, however, are 



far from satisfactory and needs considerable 
improvement, according to the study. For example, 
respondents commented that the discussion of 
theoretical framework for each subject, knowledge of 
other ERP software (other than SAP R/3), basic 
concepts of data communications, database and ^sterns 
analysis, and, knowledge of technology, configuration 
and administration of the system, are not adequately 
covered in the course. 

However, respondents rated coverage of SAP R/3 
modules, integrative nature of the course, and concepts 
of business processes and hands-on component of SAP 
R/3 system as good. There are, however, significant 
differences between the international students and local 
students. For example on the issue of covering concepts 
of business processes, international students are far less 
satisfied than the local students. The lack of functional 
experience of the majority of international students may 
be a factor contributing to this perception. It is difficult 
to cover the basic process knowledge as well as software 
skills in a particular unit exhaustively, given the 
constraints of time (36 hours of tuition) and limitation of 
the academic staffs competence on the software. 
Students are provided basic introductory knowledge of 
the appropriate business processes and software skills in 
the course and are required to develop on their own 
and/or by taking other electives that impart deeper 
knowledge and skills on process concepts and 
management. 

On design of assessment tasks, there are significant 
differences between international and local students. For 
example, international students feel that oral 
presentation and written research paper have significant 
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TABLE 2 

OVERALL PERCEPTIONS OF STUDENTS— LOCAL VS. INTERNATIONAL STUDENTS 



Perception of Students 


Local Students 


International Students 


Significance 


Overall Rating 


Curriculum design 


4.03 


2.92 


0.000 


3.59 


Course delivery 


3.51 


3.24 


0.015 


3.40 


Resources & Support 


2.71 


2.10 


0.719 


2.47 


Teaching performance 


3.01 


3.16 


0.325 


3.07 


Course materials 


2.78 


2.88 


0.101 


2.82 


Overall program 


3.30 


2.93 


0.002 


3.15 



proportion of weightage which could disadvantage them 
in view of their non-English speaking background. 
Another issue raised by them, is the field project. Field 
project generally requires students to obtain access to an 
organization and involves collecting information and 
interviewing managers. International students found it 
extremely difficult to gain access for carrying out such 
projects and generally felt disadvantaged. 

Course Delivery 

Course delivery is a challenging issue, especially, when 
a complex software such as SAPR/3 system need to be 
taught along with the theoretical concepts of the 
processes. In addition, it also depends upon various 
aspects of teaching such as knowledge of acadanic staff 
(both process and software), structure and sequence of 
the topics, design of and feedback on specific 
assessment tasks, communications, tutorial sessions and 
help on the system, availability of help and guidance, 
and quality of course materials. The overall perception 
of this course is significantly low. 

The difference between international and local students 
is not significant on the overall dimension. However, 
issues such as knowledge of academic staff, quality of 
course materials, structure of the lectures, and concern 
for learning needs, international students are 
significantly less satisfied than the local students. On 
issues such as availability of individual help, actual 
delivery of lectures, help during tutorials and access to 
academic staff, there are no differences and generally 
the respondents are more than ‘satisfied’. There are, 
however, some dissatisfaction expressed about the 
specific lecturers on the issues such as feedback on 
assessment tasks, structure of the lecture & topics, and 
general lack of concern for the learning needs of 
students. In addition, the knowledge and skills of 
academic staff is viewed as just ‘inadequate’ by the 
students and generally reflects the limitations of the 



delivery of such courses. It is observed in qualitative 
comments that the academic staff is not fully equipped 
to impart knowledge of SAP configuration. 

Support and Facilities 

Support and facilities are the major issues for all the 
respondents. Generally, respondents expressed complete 
dissatisfaction on the support and facilities provided by 
the university in mnning such IT-based and relatively 
new course. It appears that the administrators have 
failed to provide adequate support to the program and 
neutralized the positive perceptions of the curriculum 
design and delivery. 

Availability of the SAP R/3 system for practice and 
teaching, its speed and reliability are critical factors for 
the success of this program. Study reveals that the 
university has generally failed to provide such reliable 
service and availability of system to the students during 
the class as well as after ttie class for practice. Some of 
the reasons cited for such failure include lack of proper 
support from the university information technology 
services division, reluctance of ITS to take the 
responsibility of maintaining the SAP R/3 system, 
dependence on the QUT’s SAP Application hosting 
centre, limited system access to academics granted by 
the Application hosting centre (Queensland University 
of Technology) for fixing system problems and lack of 
advanced skills by the academic staff to fix some 
configuration problems, and inadequacy of the SAP 
IDES system for teaching purposes. While some of 
these problems are internal and can be resolved by the 
course administrators, others are beyond their control. 
For example, alternative options to own and maintain 
SAP R/3 system within the university, though feasible 
and resolve most of the problems, is an expensive 
solution and university is not prepared to commit such 
huge amount of resources for running a complex system 
such as SAP. 
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Availability of text books and adequacy of course 
materials provided in the course are another issue. 
Course materials/manuals prepared from the SAP R/3 
training manuals are distributed to the students. As 
expected, these materials are not comprehensive, do not 
include discussion of concepts and explanation of the 
terminology in detail, and do not provide enough 
information for students to learn on their own. Though 
the manuals are not intended to be self-study materials, 
and are to be accompanied with appropriate 
explanations in the class, students generally felt 
inadequate with these materials. They ccmplain that 
there is insufficient material to study in order to gain a 
deeper understanding. Moreover, these manuals are 
edited versions of SAP training manuals and are not 
suitable for university teadiing. It is important to 
address these issues and develop appropriate course 
materials suitable for teaching and learning at the 
university level. This condition is expected to improve 
with the availability of online materials from this year 
onwards. 

Since it is relatively a new course, not many reference 
books are available in the library. Thought it has 
improved lately, students are still dissatisfied with the 
non-availability of sufficient copies in the library. 
Secondly, most of the books prescribed in this course as 
additional reading materials are professional reference 
books and are quite expensive for students to buy. With 
a new SAP R/3 version coming every year, students are 
reluctant to invest on such expensive books which could 
become obsolete in a year’s time. General unavailability 
of the resources in the university library compounded 
this problem. 



Knowledge and Skills 

Students are asked to rate their own skills and 
knowledge gained during the course and their response 
is summarized below. The difference between the local 
and international students is again significant on ABAP 
programming, application modules and business 
processes. The overall perception however is generally 
low as shown below. Students reported to have gained 
sufficient knowledge of business processes and 
application modules. However, it appears that the 
students could not gain adequate skills and knowledge 
in systems administration, software configuration, 
implementation, ABAP programming and reengineering 
in the course. 

It is important to note that this is a business course 
designed with an emphasis on business processes and 
cross-functional integration rather than pure technical 
skills typically offered in computer science or 
information technology courses. Therefore, it is 
expected that they will be less competent in technical 
areas. International students also felt relatively less 
competent on business process skills, ABAP 
programming and application module software skills 
than the international students, according to the study. It 
may be generally attributed to the lack of functional or 
business experience of a majority of international 
students. It is difficult to gain competence in business 
processes with no prior knowledge or experience of 
business environment. The generally low overall rating 
for students’ knowledge may be attributed to the factors 
such as teaching competence, complexity of the 
software and lack of appropriate course materials. 



TABLE 3 

OVERALL PERCEPTIONS OF STUDENTS— LOCAL VS. INTERNATIONAL STUDENTS 



Skills and Knowledge 
Gained in the Course 


Local 

Students 


International 

Students 


Significance 


Overall Perception 


Application modules (SD, PP, MM, >i/CO, 
HR etc.) 


14.76 


17.49 


0.000 


3.16 


ABAP programming 


3.10 


2.15 


0.000 


2.71 


Business processes 


17.08 


15.25 


0.001 


3.27 


System administration 


2.24 


2.29 


0.258 


2.26 


Implementation & configuration 


2.78 


2.80 


0.078 


2.79 


Process reengineering 


2.91 


2.75 


0.024 


2.84 
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Competence of Academic Staff 

One of the fundamental requirements for the continuous 
delivery of the program is the software competence of 
academic staff. Though the training is provided free of 
cost by SAP, not many academics are interested to 
undergo software training. Therefore, the depth and 
range of academic expertise available for teaching in 
this program is limited. The need to continuously update 
their software knowledge with every change in the 
version through training and practice is placing 
enormous demands on academics’ time and the interest 
is waning. The complexity of the software, lack of 
practical experience of the academics in the 
configuration and implementation of the software and 
the limited expertise of academics, is contributing to the 
poor rating of the teaching performance and general 
overall impression of the program. 

Though an attempt is made, experience by this 
university suggests that it is difficult to successfully 
integrate a complex and dynamic software like SAP R/3 
into university curriculum and impart comprehensive 
software skills, process knowledge and cross-functional 
skills to students. By joining the University SAP 
application hosting centre as a member, software access 
has improved considerably and reduced the 
administrative workload for the academics. 

LIMITATIONS OF THE STUDY 

The emphasis throughout the evaluation process has 
been on the learner and how satisfied he/she is with the 
program in terms of its design, and delivery. Though 
this approach adopted may not be the best, but, given the 
circumstances, this approach traditionally serves better 
in obtaining feedback from the studaits. However, 
needless to say that the value of their feedback 
significantly depends upon the questionsasked. In order 
to ensure that right questions have been asked, several 
instruments used in the past by the Universities, 
literature on curriculum design and a focus group 
discussion with the students are used to develop these 
questionnaires. 

Some of the weaknesses inherent in a questionnaire 
survey are applicable to this evaluation also. However, 
an attempt is made to overcome some of the weaknesses 
by discussing issues with the students informally, and by 
asking their general opinions on the aspects which are 
done well, and aspects which needs improvement. In 



addition, before the administering of the questionnaire, 
students are asked to indicate their opinions and 
perceptions on the overall program, without being 
hindered by the structured questions. In fact, students 
responded well with this request, and given many other 
comments on the program, and suggestions for 
improvements. All these qualitative remarks are 
incorporated in the analysis while interpreting the data. 

CONCLUSIONS 

Universities must move from a position that emphasizes 
improved teaching to one that emphasizes improved 
learning by helping students to enhance their abilities to 
engage the learning challenges in their environment 
Though we teach the importance of aoss-functional 
teams, responsiveness to customer needs, and 
continuous real-time feedback for fine-tuning 
organizational performance in our courses, we do not 
demonstrate by example. Students often see narrowly 
functional teaching, a large bureaucracy that does not 
change curricula for years or design new ones to suit 
changing industry/business needs, and one 
end-of-course evaluation process that appear to have 
little relevance to their lives. This course being an 
innovative full-fee paying course with increasing 
demand and incorporates complex software such as SAP 
R/3 system, it is important to evaluate the course design 
and delivery and improve. Like any other business 
enterprise, universities/educational institutions also face 
timely, perishable opportunities for delivering the 
requisite skills to our students. If we fail to learn these 
new technologies and use them effectively in teaching 
and learning, we will keep ourselves as well as our 
students in the past, while the rest of the world moves 
on. This small attempt by the University to design 
business curricula with process orientation and use SAP 
R/3 system as a teaching and learning tool can be 
continued with the help of this feedback. This could 
become a crucial ingredient in developing future 
policies on course designs and delivery at the higher 
education institutions. It is expected that this process is 
continuous and improvements will be made on 
consistent and continuous basis. 
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INTEGRATING STRATEGIC 
MANAGEMENT WITH INFORMATION 
SYSTEMS CORE CONTENT 



Richard T. Grenci 
John Carroll University 



Core information systems (IS) courses in undergraduate 
business curriculums assume varying forms and cover 
varying topics (O’Hara and Stephens 1999). Despite 
differences in content and pedagogies, many 
implementations of core IS courses make use of 
traditional texts, most of which include chapters on 
hardware, software, networks, etc. Detailedcoverage of 
technology topics — especially when detached from the 
context of other business disciplines — can alienate non- 
IS students in particular and can contribute to 
perceptions of irrelevance. In that respect, it has been 
argued that a survey of information technologies can be 
inappropriate for a varied population of graduate 
business students (Novitzki 1997). Similar arguments 
have been made at a more general level, with academics 
as well as practitioners ultimately calling for an 
integrated business core curriculum as one solution that 
would serve “to demonstrate the interrelatedness of 
various business functions and how they work together 
in the firm” (Pharr 2000, p. 20). 

This discussion considers one undergraduate business 
curriculum that integrated an IS core course with a 
strategic management core course, thus framing the IS 
topics within the context of business strategy. The 
integration took place when all of the core courses in the 
junior level business curriculum were redesigned into an 
Integrated Core Experience (ICE). During the redesign, 
it was recognized that the IS course would be better 
positioned if moved from the second semester to the 
first where it could provide an introduction to business 
systems. It also was recognized that the strategic 
management course — traditionally taken in the senior 
year of study — could provide a complementary 
perspective to this introduction. As a result, “Strategy 
and Systems” became the lead course of the integrated 



business core. Perhaps more importantly, the course 
was implemented and delivered in a successful manner. 
The course description follows. 

“Strategy and Systems” introduces a model of 
the strategic management process and examines 
the organization and its environment as a 
system. The concepts of strategy and systems 
are a critical foundation for the Integrated Core 
Experience. Specific topics include: the trans- 
formation of business, systems thinking and 
process management, industry transformation 
and process modeling, organizational analysis 
and the role of the general manager, strat^ic 
thinking and information systems, environ- 
mental analysis, core capabilities and core 
rigidities, the strategic management model, 
writing for results, the value chain, 
organizational architecture, information 
technology architecture, and strategic and 
systems change. 

A review of relevant literature is used here to examine 
factors that underlie the success of such an integration of 
courses. In this case, a successful implementation might 
be attributed not to the instruction nor to cross- 
functional expertise, but rather to the particular courses 
that were integrated and the dynamics of the particular 
program and school within which they were integrated. 

COURSE CONTENT 

“Strategy and Systems” covered much of the same 
material that was covered originally by the IS course, 
but the content now was framed by a strategic 
perspective delivered in a team-teaching environment. 
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The main topics on business transformation, value 
chain, organizational structures, competitive advantage, 
etc. were organized within a framework of strategy 
themes and were introduced primarily by the 
management faculty with the assistance of articles 
and/or business school case studies. The IS faculty 
followed with coverage of relevant technology concepts 
such as process modeling, enterprise software, network 
computing, and emerging technologies, which were 
mapped to the strategy themes. This team-teaching 
environment was very synergistic and it aided in the 
smooth execution of the course as described in the 
following paragraphs. 

Complementary Topics 

Strategic Management and Information Systems are 
complementary in several ways. From the strategic 
management perspective, it has been argued that a 
technology management course should directly follow 
an introductory management course in providing a 
foundation for the rest of the business curriculum 
(Thompson, Purdy, and Fandt 1997). From the 
information systems perspective, the most covered topic 
in undergraduate IS core courses is “Information 
Systems in Organizations” (O ’Kara andStephens 1999). 
As noted, the IS topics in “Strategy and Systems” were 
organized relative to their role in supporting the 
strategic management of the firm. In addition to 
providing a more explicit managerial context, the 
broader thematic grouping of multiple IS topics 
provided for a cohesiveness and flow to the delivery of 
the frequently detailed IS topics. Furthermore, with 
management students preferring more general coverage 
of IS topics and IS students preferring more detailed 
coverage (Lee and Townsend 1999), the preferences of 
both groups of students could be accommodated by an 
intermixing of these approaches. 

Common Material 

Arguments have been made that the delivery of 
integrated content comes with a greater need for cross- 
functional expertise (Pharr 2000). However, the method 
of delivery in “Strategy and Systems” did not rely upon 
cross-functional expertise. Instead, the course content 
was geared towards a common set of readings that 
provided for common points of discussion for both 
strategy and IS. The classes were three hours in 
duration so that many of them could be split into two 
segments — the first for strategy and the second for 
systems. In essence, many of the sessions became 



separate classes taught back-to-back, ultimately 
providing different perspectives on the same topic using 
the same readings. The delivery arrangement enabled 
the IS faculty to focus on their technical expertise as the 
management faculty presented the strategic content. 
Such a delivery arrangement also can lead to a desirable 
efficiency with integrated curriculums in general — the 
elimination of redundancies (Pharr 2000). In fact, the 
integration within “Strategy and Systems” resulted in 
time reductions for both sides such that the integrated 
curriculum was able to reallocate the distribution of 
course hours. 

PROGRAM DYNAMICS 

Content-related dynamics were enhanced by various 
success factors that can be attributed to the environment 
within which the Integrated Core Experience was 
implemented. To begin with, the existing core 
curriculum had been “blocked” for several years such 
that students started the business program each year en 
masse with groups of students being assigned to a 
common set of core classes. In addition, the business 
school dean’s office had instituted policies that fostered 
stacked course loads and release time. These and other 
factors contributed to the success ofthe integration with 
respect to both scheduling and incentives. 

Time Commitment 

With respect to the students, ILonger blocks of time are 
necessary for integrated classes, thus making it difficult 
for part-time students to participate (Pharr 2000). 
However, all of the business students in the ICE 
program were full-time students. In addition, previous 
years’ students had been blocked into common sets of 
core classes. In fact, the previous blocked faculty had 
engaged in loose coordination between the courses. The 
blocks also prevented the faculty from having to offer 
multiple versions of a class to accommodate staggered 
sets of students during a transition phase. With respect 
to the faculty, large amounts of time are required for 
organizing and integrating the course content (Schaller, 
Cavarkapa, and Onge 2000). Such time commitment 
can be contrary to faculty incentive structures (Pharr 
2000). In the case of the ICE program, senior 
faculty — mostly full professors — initially designed the 
integrated curriculum and then piloted it on one block of 
students, thus breaking ground for the rest of the 
faculty — including untenured professors — for the full- 
scale redesign and implementation. 
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Supportive Administration 

The administration is a key factor in the success of an 
integrated curriculum effort (Pharr 2000). In fact, one 
of the faculty members who piloted the ICE program 
was the dean of the business school, thus helping to 
make the integration a top-down effort. Perh^s more 
importantly, the business school administration had 
instituted scheduling policies that fostered a modular 
curriculum structure. As such, core courses gave way to 
core modules that created concentrated periods of 
teaching time (sometimes twelve or more hours per 
week in a traditionally six-hour per semester load) offset 
by down time that could be dedicated to other scholarly 
activities. Ultimately, the administration provided both 
flexibility and incentives for the integration effort. 

CONSIDERATIONS 

Although several factors combined to help make the 
integrated “Strategy and Systems” course a success, 
some factors may need to be reexamined. 

Grading 

Due to the readings-intensive format of the course, 
participation was weighted at 30% of the final grade. 
This compares with an average for IS core courses of 
around 10% (O’Hara and Stephens 1999). The implica- 
tions of this weighting need to be investigated. In fact, 
the implications of the weight on the integrated final 
exam revealed possible problems. At 25% of twelve 
credits all riding on aie exam for the entire integrated 
core, the stress on the students became apparent. With 
respect to grading, some have argued that an integrated 
grade provides motivation for students to do well across 
all of the subjects (Schaller, Cavarkapa, and Onge 
2000). However, the ICE program resulted in a 
blending of grades such that most students ended up 
with an average grade and very few earned A’ s. 

Systems Development Topic 

More than half of undergraduate IS core courses utilize 
a project (O’Hara and Stephens 1999). Moreover, it has 
been argued that a systems analysis project can be an 
effective tool in the IS core course (Harder 1999). 



However, given the integrated nature of the course 
content, it is unlikely that “Strategy and Systems” will 
be able to incorporate such a project. In addition, 
systems development, the second-most covered topic in 
IS core courses (O’Hara and Stephens 1999), was not 
covered in “Strategy and Systems.” 

Although “Strategy and Systems” was smoothly 
executed, it has yet to be determined if the integrated IS 
content is more effective than the previous version of 
the IS course. Anecdotally, the feel of the integrated 
course seemed positive, and student evaluations of the 
IS instructor were more favorable with respect to the 
integrated course as compared to previous versions of 
the IS course. Further examination of “Strategy and 
Systems” is necessary. 
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A WIN-WIN PROPOSITION FOR 
TELECOMMUTER CENTERS: 

A SUCCESSFUL ACADEMIC AND 
CORPORATE PARTNERSHIP FOR 
RURAL TELECOMMUTING IN NEW ENGLAND 



Charlotte S. Stephens 
Louisiana Tech University 



ABSTRACT 

A grant funded by the Center for Entrepreneurship and Information Technology Grant (CEnIT) at Louisiana Tech 
University led to the exploration of successful rural telecommuting within an established investment firm which 
employs over four hundred telecommuters in New England. A successful program for almost a decade, the firm 
places telecommuter centers in academic environments. Not only do these centers host the network and provide 
equipment, technical, and managerial sipport for telecommuters, they employ students on a part-time, flexible basis 
to serve as agents and to develop online learning modules from specifications. Student teleworkers are a critical 
success factor for the Virtual Office Environment (VOE) at the investment firm. This research-in-progress paper 
reports on studies conducted at three telecommuter centers and future research. 



INTRODUCTION 

The U.S. Labor Department reports that for the past 
decade, about half of full-time students work part-time. 
These jobs are rarely valuable lines on their resumes. 
For example, students in Analysis and Design classes 
often choose restaurant systems because restaurants are 
a major employer of students. Students may choose 
internships for more valuable short-term, career-oriented 
work experience. However, a well-established New 
England investment firm, headquartered in Boston, is 
providing part-time telework jobs for students in a 
professional environment with flexible hours. Further- 
more, they are providing professional training for the 
students including Professional Skills Workshop, 
Business Etiquette/Professionalism, Communication 



Styles/Skills, Resume Building, and Interviewing Skills. 
Supervisors assign students to teams and evaluate team 
performance. According to the incubator site for this 
innovative telework application, “students process work 
ranging from a simple address update to million dollar 
cash investments. . . students are expected to adhere to 
high quality standards . . . management staff provides 
quality training and technical support to the students 
daily.” In many ways, the four colle^ locations 
resemble satellite offices staffed with full-time 
supervisors and part-time student employees. Many 
students join the firm full-time after graduation. Their 
work experience is a valuable technology component of 
their college education, and several students interviewed 
indicated that this work experience made them better 
students. 
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BACKGROUND 

While the rationale for telecommuting would appear to 
be a win-win proposition, particularly after September 
1 r*', telecommuting has grown at a much smaller pace 
than forecast a decade ago. Resistance to telecom- 
muting has been strong, especially among managers. 
The telecommuter center should ameliorate many of the 
problems experienced 1^ telecommuters and their 
managers. Instead of a long commute to an urban 
environment, an employee could drive to a satellite 
office or telecommuter center where support and 
supervision would be provided. The telecommuter 
center offers technical and supervisory support. 
However, telecommuter centers have experienced high 
failure rates because they are not a viable business 
model. 

CEnIT GRANT SUMMARY: 
TELECOMMUTER CENTERS 
IN RESTORED HISTORIC 
PROPERTIES OF NORTH LOUISIANA 

Telecommuter centers would provide a work infra- 
structure for local employees of organizations located 
outside the North Louisiana commuting area Centers 
located in restored historic properties offer special tax 
advantages for participating organizations. Technology- 
related jobs would be made available to North Louisiana 
residents, including Tech graduates who wish to remain 
in Louisiana. USA Today (10/17/01, page Bl)reports 
that “terrorism boosts demand” for telecommuting and 
distributed work sites, helping to eliminate central 
points of failure. Both the time and technology are right 
for an increase in telecommuting. This five-phase 
project addresses the CEnIT mission of “entrepreneur- 
ship development in technologically under-developed 
communities” and its thrust area is Technology 
Entrepreneurship. The project is in accord with the 
College of Administration and Business’ Strategic 
Charter, in which citizens of North Louisiana are 
stakeholders. 

CRITICAL SUCCESS FACTORS: 
CURRENT FINDINGS 

In Maine and Vermont, these satellite offices are also 
telework or telecommuter centers, supporting not only 
student employees but over four hundred 
telecommuters. While teleworkers have not become as 
large a percentage of the workforce as forecasted a 
decade ago, the firm has operated a successful telework 



program, called the Virtual Office Environment (VOE), 
for a decade. The telecommuter centers located at two 
colleges in Maine and one in Vermont have been a key 
to the success of the telecommuter program. The 
partnership has required the continuing involvement of 
all stakeholders. The firm’s CEO was the graduation 
speaker at the Vermont location. A group of student 
employees visited the Boston headquarters and observed 
agents working in different career paths. The Assistant 
Vice President for the Virtual Office Environment in 
Maine is often a guest speaker in business classes. In 
Vermont, the academic partner is a leader in online 
learning. There, the firm contracts with their experi- 
enced course designers and online course developers, 
and employs part-time students to work with these 
professionals to develop online learning modules for 
telecommuters. The partnership is multidimensional 
and has been a win-win for all stakeholders: part-time 
student employees, full-time telecommuter employees, 
employees in the corporate office, the firm, the states, 
and the colleges hosting satellite offices/ telecommuter 
centers. Top management at the firm has provided 
unwavering support for the Virtual Office Environment. 

Part-time student employees and the hosting of the 
center in an academic environment have made 
telecommuter centers a viable business model while at 
the same time making work at home more successful. 
These centers support equipment and the network for 
telecommuters, provide training and technical support, 
are beginning to provide supervisory support for 
telecommuters, and act as a back-up facility should 
home operation be impossible. High speed, secure, 
dedicated telecommunicatiais link the centers to 
Boston. Telecommuters link to their respective centers 
via ISDN or cable. Further, these centers act as a back- 
up facility for the headquarters, with student part-time 
employees again playing a vital role. When 
headquarters has a fire drill, for example, the firm’s 
operations cannot be halted. Instead, tasks are routed to 
the telecommuter centers which will be fully staffed 
during such times and to telecommuters. When 
additional hours are needed for peak periods, students 
may choose to work additional hours during the 
weekend and the firm does not incur overtime pay. 
Students, on the other hand, participate in a Profit 
Sharing Plan and 401(K) plan. 

Students working in a Virtual Office Environment learn 
to use virtual management tools, tools which are so 
effective that most have been adapted in the traditional 
environment at Boston headquarters. Initially, only 
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exceptional and experienced employees were allowedto 
work at home. With ttie expansion to Maine and 
Vermont, new employees were telecommuting. The 
investment firm wanted to insure that these new agents 
had all the tools as agents at headquarters. Self-paced 
training modules developed for these teleworkers were 
so effective that they are now used at Boston 
headquarters as well. Lotus Notes supports 
asynchronous collaboration, but if an immediate need 
arises, a keystroke initiates a pop-up window which 
overlays the supervisor’s screen and may include others 
as well. Anyone receiving the request for help may 
reply with the needed information. This Virtual Office 
Environment tool has also been adopted in the 
traditional work environment. With appropriate virtual 
management tools, anything done at a satellite office 
could be done at a satellite office at home, says the 
Vermont manager. Not all management is virtual, 
however. Relationship management remains critical. 
“Phone calls, e-mails only go so far when one is 
frustrated.” The social aspect cannot be assumed; it 
must be managed with scheduled meetings, socials, 
visits to the center, one-on-one feedback. 

The researcher visited three college sites, talked with 
students, interviewed the Assistant Vice President in 
Maine and Vermont, and a supervisor for Maine. 
Students working in the centers were also observed. 
Students must learn to conduct themselves as business 



professionals. Although students usually walk to work 
from their dorm or apartment, th^ are expected to 
adhere to a relaxed professional dress code. In one 
center, a sign at the entrance gently reminds students 
that tank tops are not professional dress. The firm 
requires a professional demeanor for student agents. 
One student told the researcher that working for the 
investment firm has made her a better student because 
she has become more organized, more disciplined, and 
more “detail oriented.” The Assistant Vice President in 
Vermont describes his job as part guidance counselor. 
“We have to build the corporate mentality into students 
so that they work without gossip or childish behavior. 
They have to learn to work cooperatively. Students are 
assigned to one supervisor who works with them 
individually.” Students have incentives to reduce errors: 
time off with pay. Their performance as individuals and 
as a team is tracked and evaluated. According to a 
brochure published by the firm, student telework 
benefits include the following: team environment, 

hands-on technology e;q)erience, extra income, 
investment industry exposure, technical computer skills, 
career opportunities, flexible workhours, profit sharing 
and 401K. 

Future plans for research are detailed in Appendix A. 

Working references are available from the author 
upon request 
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APPENDIX A 

FUTURE RESEARCH PLANS 



Original Proposal 


Continuation Proposal 


Phase 1: Develop critical success factors for successful 

telecommuting centers by conducting in-depth 
case studies at successful centers. 


Complete and reviewed by Board. 


Phase 2: Identify candidate historic sites in north 

Louisiana using these criteria. Students and 
professors from the following fields would 
become involved in this process: Architecture, 
Civil Engineering, History, English, Marketing, 
Accounting, Computer Science, and Computer 
Information Systems. 


Since the successful model uses an academic 
environment, candidate sites are limited. 
Recruiting participating companies has a higher 
priority. 


Phase 3: Develop a marketing portfolio of properties, 

included both a paper brochure and a web site. 
The portfolio would include tax credit 
information for historic restoration, a history of 
the building and surrounding area, featured area 
attractions, and architectural rendering of the 
restored property as a telecommuter center. The 
properties would be mapped to criteria for 
successful centers. 


Instead of a marketing portfolio of properties, the 
continuation proposal would deliver a multimedia 
resource guide to market the concept of a telework 
center for part-time student employees and full- 
time telecommuters. Completed academic research 
would also be published. 


Market the use of restored historic p"operties in North 
Louisiana as telecommuter centers, using research on criteria 
for successful telecommuter centers and the portfolio of 
properties. Companies who currently hire Louisiana Tech 
graduates would be included. 


State, local, and university officials would use the 
grant’s research deliverables and the multimedia 
resource guide, as well as special incentives, to 
gain participation in telework programs. I would 
serve as a resource person. 


Serve as an incubation center for the first three centers, 
working closely with the state and companies hosting 
Louisiana telecommuters to provide IT infrastructure needs, 
skilled employees, student interns, management consulting, 
tax accounting consulting, etc. 


Participating companies would provide supervisors 
and a corporate infrastructure. I would continue to 
serve as a resource person and to conduct research 
on successful telework practices, possible in a 
Telework Research Center within CEnIT. 
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ABSTRACT 

The paper introduces a technique for teaching students how to create an object-oriented design. The technique has 
the advantage that it provides students with an (almost) automatic way of starting the design process. We also 
present a preliminary evaluation of the technique, based both on the students' performance on an object-oriented 
design project and on their perceptions of the technique, as assessed through a brief questionnaire. 



INTRODUCTION 

Object-oriented design techniques, in the fonn of say 
E-R diagrams (Chen, 1976) or Abstract Data Types 
(Liskow & Zillis, 1 974), have been around for more than 
25 years. However, it is only recently that 
object-oriented design has become so mainstream that it 
is included in textbooks on systems design and analysis 
(e.g., Hoffer et al, 2002; Whitten et al, 2001). Most 
textbooks give excellent introductions to the various 
object-oriented design techniques and notations, and 
outstanding explanations of object-oriented concepts, 
such as classes, instances, attributes, behaviors and 
inheritance. Unfortunately, to our knowledge, none of 
the textbooks provide any guidance on how to go about 
creating object-oriented designs. As a result, students 
often report being at a loss when it comes to tackling an 
object-oriented design project 

To help students overcome this problem, the first author 
developed a technique for teaching students how to 
create an object-oriented design based Rumbaugh et al 
(1991). The technique has been used in a number of 
courses. In this paper, we focus on two, namely a 
freshman level course entitled “Introduction to 
Information Technology,” offered to students enrolled 



in a baccalaureate program in Information Technology 
(the “IT course”) and taught by the first author, and a 
Junior level course entitled “Systems Analysis and 
Design,” offered primarily to students pursuing a BBA 
with a specialization in Information Systems (the “IS 
course,”) and taught by the second author. In this paper, 
we briefly introduce the teaching technique, and then 
present a preliminary evaluation of the technique, based 
both on the students’ performance on an object-oriented 
design project and on their responses to a brief 
questionnaire. 

THE TECHNIQUE 

We start by giving students a conceptual introduction to 
object-oriented design and give an in-depth explanation 
of object-oriented terminology (classes, instances, 
inheritance, message passing, and so on). We then teach 
students that creating an object-oriented design consists 
of four phases, namely 

1. Identification of an initial set of classes; 

2. Identification of the attributes and behaviors 
identified in Step 1; 
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3. Identification of any super- and subclasses; 

4. Specification of the behaviors in of the classes 
identified in 3. 

We also suggest that students start the first step of the 
design by simply identifying all nouns in a problem 
statement. They are then taught to eliminate: 

1. Redundant nouns (i.e., synonyms); 

2. Nouns that obviously refer to attributes and 
behaviors; 

3. Nouns that concern the implementation of the 
system to be designed (e.g., such nouns as “the 
system,” “information,” “record,” etc.); 

4. Nouns that are irrelevant because they either 
concern classes of which the system does not have 
to maintain any information (e.g., the “user”) or 
because they occur only in a section of the problem 
statement that state explicitly that the software 
under development does not have to display a 
particular piece of functionality. 

The list of nouns left after elimination is the initial list 
of classes. Students are also taught not to eliminate a 
noun if they have any doubt as to whether to eliminate 
the noun in question. The rationale is that it is easier to 
eliminate an irrelevant class later on than to introduce a 
new class late in the design process. 

It is important to note that the initial step in the first 
phase of the design process can be performed (almost) 
automatically as it simply involves identifying the nouns 
in a problem statement. This makes it easier for 
students to get started, and sets the technique apart from 
other design strategies. The first stage is, howeva*, 
often tedious as it involves the creation of a long list of 
nouns, most of which are eventually eliminated. Many 
students are tempted to take shortcuts. We strongly 
discourage them from doing so. 

Many of the nouns that were eliminated in the first stage 
of the design process become important in the second 
phase, as they were eliminated because they were 
attributes. We also tell students that, unlike relevant 
classes, attributes can often be derived from knowledge 
that one has about the application domain, and therefore 
do not necessarily appear in the problem statement. 
However, we suggest that students be conservative when 



it comes to introducing new attributes and to only 
introduce attributes when they are certain that they are 
necessary. After all, attributes can easily be introduced 
later on in the design process while the introduction of 
irrelevant attributes often leads to unnecessarily 
complicated design. 

Students are also taught that for every attrfcute 
identified, one typically needs two behaviors, namely 
one to set or modify the value of the attribute and one to 
access the value of an attribute. They are also taught to 
look for verbs in the problem statement as they typically 
refer to something that the application has to be able to 
do and therefore often correspond to a behavior. 

The third phase of the design process is the 
identification of super-classes. Students are taught to 
look for classes that have a significant number of 
attributes in common and to introduce a super-class if 
two or more classes share a significant number of 
attributes and behaviors. 

The final stage in the object-oriented software design 
process is the specification of the behaviors of the 
classes and super-classes identified in stages 1 and 3. 
However, since we only covered this aspect of the 
design process only in the IT course, we will not 
elaborate on this aspect of the design process here. 

We covered the material in three class sessions of 50 
minutes. However, we also wrote extensive notes on the 
techniques and made those available to our studentsover 
the Internet. At this stage, it is important to report that, 
while class attendance in the IS class was good, class 
attendance in the IT class was poor. There may be a 
number of reasons for this. First, since the material had 
been made available to students beforehand, students 
may have been of the view that they did not need to 
attend class. Second, many of the students enrolled in 
the IT course were in fact seniors needing the credits 
that they would earn by taking this course to graduate. 
These students may not have been the most motivated of 
learners. 

EVALUATION 

As noted, students typically complain about being at a 
loss how to start a design project. Since our technique 
allows students to start the design process almost 
automatically, we hypothesized that it would enable 
students to create better designs and would therefore 
positively received by students. In order to test these 
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two hypotheses, we performed two types of evaluation. 
First, we gave all students a design project. Second, we 
administered a questionnaire asking students about their 
opinions about the technique. 

In order to evaluate the effectiveness of our teaching 
technique, we gave all students the design project 
enclosed in appendix A. Those enrolled in the freshman 
IT course were asked to create a full object-oriented 
design, including a full specification of all the behaviors 
that they had identified. The assignment was worth 1 0% 
of the final grade. Those enrolled in the IS course were 
asked to identify classes and their attributes and 
behaviors, and any relevant super-classes. They were 
not asked to specify the behaviors. Their projects 
consequently contributed only a nominal amount to the 
overall grade. 

The student mistakes could roughly be divided into the 
following five categories, namely 

1 . Failure to identify the appropriate classes Customer 
and Vehicle; 

Example: Some students identified a class Customer 
Contact Information. 

2. Failure to identify appropriate sub-class super-class 
relationships; 

Example: Some students identified Vehicle as a 
sub-class of Customer. 

3. Failure to identify the appropriate attributes and/or 
behaviors and/or the association of an attribute or 
behavior with an inappropriate class; 

Example: Some students associated the behavior to 
determine whether a vehicle was fast or 
fuel-efficient with Customer, rather than Vehicle. 



4. Failure to identify the appropriate sub-classes; 

Example: Some students did not identify Pickup, 
SUV etc as sub-classes of Vehicle. Instead, they 
included an attribute type in Vehicle to distinguish 
between the different sub-classes. 

5. Failure to use inheritance in any way; 

Example: Some students, having identified the 
appropriate sub-classes, re-specified all attributes 
and behaviors within the sub-classes. 

Obviously, the errors are not independent. For example, 
a student can only display erra* 3 if they have correctly 
identified the classes Customer and Vehicle. We used 
this observation \\iien compiling the table below. A 
student would only be marked as having displayed error 
2 if they had not made error 1 , error 3 if they had not 
made errors 1 and 2, and so on. With this in mind, table 
1 gives the frequency of errors made by the students. 

The results indicate that many students failed to fully 
grasp the design methodology that we tried to convey. 
Only one third of the students made either no errors at 
all or only error 5, and hence arrived at the correct class 
hierarchy, while about 22% fail to identify even the 
basic classes Customer and Vehicle. We also note that 
about as many students enrolled in the IS course either 
made the most fundamental error, or made no error at 
all, while the majority of students enrolled in the IT 
course made the error of failing to identify the relevant 
sub-classes of the Vehicle class. 

We also administered a student questionnaire in order to 
determine the students’ perceptions of the techniques 
taught. The questionnaire, which was administered in 
the final class session, consisted of 15 questions, with 5 
asking some demographic information (status, gender, 
major, number of classes taken previously in 



TABLE 1 

PERCENTAGE OF STUDENTS MAKING SPECIFIC ERRORS 





Total 


Error 1 


Error 2 


Error 3 


Error 4 


Error 5 


No Error 


IT Course 


66 


9% 


1.5% 


12% 


53% 


12% 


12% 


IS Course 


50 


40% 


8% 


0% 


6% 


0% 


46% 


Total 


116 


22.4% 


4.3% 


6.9% 


32.8% 


6.9% 


26.7% 
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object-oriented techniques, and knowledge of which 
object-oriented programming languages) and 10 asking 
students about 

1 . The difficulty of identifying classes; 

2. The difficulty of identifying attributes and 
operations of these classes; 

3. The difficulty of identifying super-classes; 

4. The clarity of the Object-Oriented Design 
Techniques; 

5. The difficulty of learning the techniques; 

6. The difficulty of applying the techniques to different 
scenarios; 

7. The difficulty of the technique compared to data 
modeling (ER Diagrams); 

8. Its difficulty compared to process modeling (Data 
Flow Diagram); 

9. Its difficulty compared to logic modeling (pseudo 
code/algorithm); 

10. The overall effectiveness of the object-oriented 
design techniques. 

The questions were answered on a five point Likert 
scale, with a low score indicating that the student 
considered the techniques to be difficult. Unlike the IS 
students, the IT students could not be expected to have 
been exposed to ER Diagrams, process and logic 
modeling. Students were therefore instructed to answer 
not applicable ifthey were unfamiliar with the technique 
in question. Such answers were obviously ignored in 



the analysis below. Also, we calculated, for each 
student, his or her overall average score. Table 2 gives 
the results. 

First, notice that we had only 39 respondents in the IT 
course while 66 submitted the project, thus providing 
some evidence for our earlier assertion that attendance 
in the IT course was poor. Second, with the mid point 
on the scale at 3, we also notice that students enrolled in 
the IS course in general regard object-oriented 
techniques as relatively easy, while the students enrolled 
in the IT course are in general slightly less positive. Of 
particular note is the overall perception of 
object-oriented techniques, which is below the mid point 
for students enrolled in the IT course and above the mid 
point for the students in the IS course. The obvious 
hypothesis that this difference may be due to the fact 
that students enrolled in the IT course have not been 
exposed to other design techniques and therefore have 
nothing to compare object-oriented techniques to is not 
borne out by the data. The mid-point on question 10 for 
those students enrolled in the IT course who have 
answered questions 7 through 9, and therefore 
presumably have had some exposure to alternative 
design techniques, is 2.58. 

We also tried to see if there was any relationship 
between any of the demographic factors and their 
opinions about object-oriented techniques. Tables 3 to 
6 try to answer these questions (totals do not always add 
up to 87 as not all students answered the demographic 
questions): 

None of the above differences is statistically significant. 
There is one perhaps counter-intuitive trend: The more 
exposure a student has had to object-oriented techniques 
and the more object-oriented programming languages he 
or she knows, the more difficult he or she seems to find 
the object-oriented software design techniques. There 



TABLE 2 

ANSWERS TO QUESTIONNAIRE— MEAN POINT ON LIKERT SCALE 





Total 


Qi 


Q2 


Q3 


Q4 


Q5 


Q6 


Q7 


Q8 


Q9 


QIO 


Overall 


IT Course 


39 


2.61 


2.90 


2.77 


3.10 


3.15 


3.23 


2.94 


2.92 


2.94 


2.57 


2.91 


IS Course 


48 


3.40 


3.52 


3.33 


3.58 


3.27 


3.04 


3.42 


3.48 


3.38 


3.5 


3.39 


All 


87 


3.05 


3.24 


3.08 


3.37 


3.22 


3.13 


3.22 


3.24 


3.19 


3.08 


3.17 
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TABLE 3 

OVERALL AVERAGE BY STATUS 





Total 


Overall Average 


Freshman 


8 


2.98 


Sophomore 


14 


2.98 


Junior 


35 


Z.22 


Senior 


28 


3.24 



TABLE 4 

OVERALL AVERAGE BY NUMBER OF 
OBJECT-ORIENTED LANGUAGES KNOWN 





Total 


Overall Average 


0 languages 


8 


3.37 


1 language 


47 


3.29 


2 languages 


18 


3.01 


3 languages 


14 


2.86 



TABLE 5 

OVERALL AVERAGE BY PREVIOUS 
EXPOSURE TO OBJECT-ORIENTED 
TECHNIQUES 





Total 


Overall Average 


1 class 


36 


3.24 


2 classes 


27 


3.28 


3 classes or more 


24 


2.96 



TABLE 6 

OVERALL AVERAGE BY MAJOR 





Total 


Overall Average 


IS 


43 


3.23 


IT 


26 


3.06 


Other 


5 


3.76 



are of course many reasons that one can advance for this 
state of affairs, including interference from previous 
knowledge, the fact that often object-oriented languages 
such as Java are taught as a “bag of tricks” without any 
attempt at instilling design skills in the student as well, 
or perhaps the fact that a student who knows a few 
object-oriented languages believes that there is nothing 
to be gained from learning anything more about 



object-oriented techniques. However, at the moment, 
we have no evidence for or against any of these 
hypotheses. 

CONCLUSIONS AND FUTURE WORK 

Object-oriented software design techniques are 
obviously not some fad that will fade in the near future. 
It is therefore crucial that educators in Computer 
Science, Information Systems and Information 
Technology develop techniques to teach object-oriented 
software design effectively. The technique that we 
developed has the advantage that it allows novices to 
start creating an obgect-oriented design (almost) 
automatically. Unfortunately, while many students 
whom we taught this material do not seem to find these 
techniques difficult or hard to grasp, the quality of the 
object-oriented designs that they created was not great. 

Obviously, the study reported here is merely a 
preliminary investigation. There are many questions 
that remain unanswered. For example, is it the case that 
the poor performance of some students is due to the fact 
that they did not attend class? If so, does this mean that 
teaching these techniques in modes other than 
face-to-face is likely to lead to difficulties, especially in 
view of the fact that most students who did not attend 
class did relatively well in their other assessments, such 
as Web design projects and written examinations? Is it 
indeed the case that students who know a few 
object-oriented languages find object-oriented design 
harder than students who come to this material without 
any prior knowledge? If so, why? Also, is there any 
correlation between the quality of a design that a student 
produces and their perception about the difficulties of 
these techniques? Since our questionnaires were 
anonymous and the information obtained from them did 
not allow us to identify any students, we obviously 
could not address this question in this study. We hope 
to start providing answers to these questions in a more 
detailed study. 

It may also be the case that part of the difficulty that 
students are experiencing in creating good design is due 
to the fact that, as yet, there are no software tools to help 
them create and, p^haps more importantly, visualize 
such designs. We hope to start building such software 
support in the near future and evaluate whether and how 
it influences both the designs that students produce and 
their perceptions of the difficulty of the various 
techniques described here. In the meantime, we will 
continue using and refining the teaching techniques 
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described in this paper as we are convinced that they 
provide an excellent way of introducing students to 
object-oriented software design. 
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APPENDIX A 
THE DESIGN PROJECT 

A friend has approached you to help him build an information system to help increase business for his web-based 
second hand car dealership. He wants to maintain information about vehicles for sale and potential customers. For 
each customer, he wants to maintain their name and contact information (home telephone number, office telephone 
number, mobile phone number and address). For each customer, he also wishes to record the type of vehicle they 
are interested in (light truck, pickups , SUV or passenger car), their preferred color, whether they prefer a fast car 
or the most fuel efficient vehicle in its class, and the price range that they are looking in. 

All vehicles have a unique registration number, and are manufactured in a particular country in a particular year. 
Each car also comes in a particular color, has a particular engine capacity and a weight. The system should also 
record the type of transmission (manual or automatic) and whether the car is front-wheel, rear-wheel or four wheel 
drive. For light trucks and pickups, the system should also maintain their carrying capacity (the weight of the load 
that they can transport). 

Your friend uses the following complicated rule of thumb to determine whether a car is fuel efficient or fast 

He first divides the engine capacity weight. If the car is made in Germany, he multiplies this by .3. If 
the car is an automatic, he divides this by 1 .2. If the car is a pick-up, he then divides this by 1 . 1 . If the final 
number is over 1.3, the car is deemed to be fast; otherwise it is deemed to be fuel-efficient. 

He also uses a complicated formula to determine the price of the car: 

If the car is an import car (i.e., not made in the United States), he multiplies the engine capacity by 3; 
otherwise he multiplies it by 2.9. He then subtracts 5% for each year that the car is old if the car is a light 
truck, and 4% for any other type of vehicle. 

Your friend would like the system to be able to automatically perform these calculations. He obviously wants to be 
able to update the information that he maintains about his potential customers. Eventually, he would like the system 
to be able to automatically find vehicles from his stock of cars that might be of interest to potential customers, but 
he realizes that this is difficult and does not expect the system that he asked you to design to be able to perform this 
task. 
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ABSTRACT 

Problem-Based Learning (PBL) has been popular as a teaching and learning strategy inMedicine, Law, and business 
schools, but it is less common in Information Systems (IS) education. Action Project Learning (APL) strategy has 
also been applied in professional educations. These strategies focus on different ways to achieve positive learning 
outcomes. The authors adopted a combination of both PBL and APL in teaching IS in their respective institutions 
in the fall of 2001: one in the US and one in Hong Kong. The PBL approach uses both structured and ill-structured 
problem cases as stimulus for problem solving and learning. It fosters generic skills development of IS graduates. 
The APL approach uses real-life projects to allow students to relate theory to practice through hands-on experience. 
The authors combine these two strategies by first introducing PBL to help students to develop the generic skills 
required of an IS professional, followed by implementing APL to achieve desired learning outcomes. Appropriateness 
of subject as perceived by the students, skills development, learning effectiveness and teamwork, as well as a 
comparison between these two approaches and the traditional approach, were all evaluated through a questionnaire 
survey at the end of the semester. This paper recommends a balanced mix of PBL and APL to enhance quality of 
learning and teaching in IS courses. It reports on students^ perceptions of the two approaches and their comments. 
We also present our strategy, together with its strengths and difficulties. 



INTRODUCTION 

Rapid technological development in the IS profession 
has made it difficult for educators to cover all relevant 
knowledge in the curricula, and also, it has rendered 
knowledge getting out-dated very quickly. Professional 
competence consists of three crucial components: 
knowledge, skills and capability, and personal 
development (Johnson, 1998). Students need to equip 
themselves with the skills to meet this challenge 
(Horwood et al., 2001). The authors have applied the 
PBL strategy to teach IS courses in the past years. The 
learning activities among students have improved. It has 
been considered that PBL combined with APL will 
achieve even better learning outcomes since studentsare 
able to relate theory to practice. 



This paper reports our application of the combined PBL 
and APL teaching strategy to Systems Analysis and 
Design courses in our institutions. Problem cases and 
real-life projects had been selected for the students, who 
undertook PBL and APL through collaborative learning. 
They worked in teams of three to six. This approach is 
unique and has worked very successfully in the courses. 
In the following sections, we present the details of the 
approach we adopted, the students’ perceptions of our 
approach, and its strengths and difficulties. 

PROBLEM-BASED LEARNING STRATEGY 

The amount of knowledge in IS is increasing at an 
accelerating rate. Students need to equip themselves 
with the skills of knowing how to learn, and it is 
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important for them to become self-directed learners 
(Walker 2000). Problem-based learning has become, in 
tertiary education, an increasing^ popular learning and 
teaching strategy that provides students with problem 
solving, self-directed learning and collaborative learning 
skills (Biggs, 1999). It is also a curriculum design that 
not only develops students’ problem-solving skills, but 
also helps them to acquire other necessary knowledge 
and skills (Boud and Feletti, 1997). This approach to 
learning is more student-centered than teacher-centered 
because students assume greater responsibility for their 
own learning. Students normally work in teams to tackle 
ill-structured problems, through which they develop 
high-order thinking and problem-solving skills. The 
teacher takes the role of a facilitator instead of a 
knowledge provider as the students learn from one 
another, through the World Wide Web, and from other 
information resources. 

Given a problem case, students analyze the problem 
situation using their prior knowledge and experience. 
They then formulate hypotheses and make plans to solve 
the problems. This requires students to search for further 
information in order to understand and tackle the 
problems. In this process, students undertake self- 
learning. New knowledge is shared and exchanged 
amongst team members in such a way that the team 
practices problem-solving through collaborative 
learning. Students report their findings through written 
reports and presentations. They also have an opportunity 
to further develop the generic skills required of an IS 
professional. However, in order for PBL to be 
successful, there is a need to establish an environment to 
facilitate students’ learning. 

ACTION PROJECT LEARNING STRATEGY 

Action Project Learning allows students to learn by 
doing. It uses real-life projects as the starting point. 
Problem solving is an inherent characteristic of Action 
Learning as students practice problem solving through 
working on a project. Evidences have shown that APL 
improves quality of students’ learning and brings lasting 
improvement to teaching (Kember, 200 1 ). Students plan 
and control the problem solving processes and are 
actively involved in learning through participation. 

The use of APL provides an environment for students to 
acquire some ‘real’ project work experience and also to 
apply theories to practical situations. A unity between 
theoretical exploration and practical experience has 
proven to be successful (Daqing et al., 2001). This 



requires students to integrate their knowledge and 
experience to solve application problems. Positive 
feedback on students’ learning has been received (Shi et 
al., 2001). Given a real-life project, students need to 
diagnose what needs to be done, set goals covering their 
skills and knowledge, and then decide how the 
application problem can be disentangled to meet the 
needs of the sponsor of the project. They need to 
exercise various skills such as problem analysis, 
communication, creative thinking and technical 
knowledge. Normally, they also require new skills and 
knowledge to help them deal with practical situations. 
Having some hands-on experience will he^ them to 
assess their own competence so that they know their 
development needs and in what areas they should 
improve themselves. In general, students are motivated 
to better perform as they see their project will 
subsequently be adopted as a real, productive system. 
They work in teams through which they practice and 
appreciate teamwork. They also develop leadership and 
project management skills. 

The main issue is how to find projects for students to 
practice? Many technology application opportunities in 
a university setting can be brought into a program, 
through which students can acquire ‘hands on’ 
experience (Kazlauskas, 200 1 ). Real-life projects can be 
secured in local industries or in the university 
environment, e.g., book shop, canteen, student union etc. 
Before APL can start, a project has to be supported or 
offered by an organization or a department, who acts as 
the client. The projects should enthuse students, who 
undertake them to meet the needs of the client. The 
authors have managed to find projects for the students 
within their respective university campuses. 

COMBINATION OF 
PBL AND APL STRATEGIES 

The intention of combining the two learning and 
teaching approaches, PBL and APL, is to enhance 
students’ learning effectiveness and also to let them 
better appreciate the IS courses. Although ‘real’ or 
‘authentic’ cases are applied in PBL, students may 
perceive that they are not working on real projects as 
they may not be required to produce a product but are 
only asked to report and present their findings. In their 
minds, these are all paper work. In actual fact, the PBL 
approach builds a theoretical framework for problem- 
solving, in which students practice problem analysis, 
project planning, information search, self-study, and 
collaborative learning. Moreover, various PBL 
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processes enable students to develop the generic skills 
required of an IS professional. 

APL, on the other hand, provides students with the 
hands-on experience of learning by handling real-life 
projects. Students learn through active participation in 
real projects that involve external people. However, 
having APL alone may not be effective when students 
do not have the theoretical framework aforementioned. 
Therefore, combining the PBL and APL approaches 
seems to be desirable when a balance of time allocation 
to each of these approaches can be worked out. 

Students are given real-life projects to handle. These 
may require them to perform fact-finding by 
interviewing users for more information, determine the 
best course of actiai, make a project plan, develop the 
system and present evaluation strategies. Students go 
through the APL process as follows: 

• Students are given a brief description of the projects 
and problem situations that require investigations and 
solutions. 

• Students bid on their choice of project, which 
determines their project allocation priority. 

• Project allocations are to be made with diffaent teams 
working on different projects. 

• Students work through the projects by: interviewing 
the client (users) and drawing up the terms of 
reference of the project, determining the project 
requirements, designing the system, choosing an 
appropriate programming language and developing 
the system. 

• Finally, students give a presentation and 
demonstration to the client in the presence of the 
whole class. They comment on their learning activities 
by submitting a group report. 

STRENGTHS OF THE 
COMBINED PBL AND APL STRATEGY 

The combined PBL and APL strategy provides a 
teaching and learning aivironment that allows students 
to: 

• apply knowledge to practical situation 



• experience the integration of different subject 
disciplines 

• practice and appreciate teamwork 

• develop leadership and project management skills 

• develop the required generic skills required of an IT 
professional 

• develop self-acceptance 

• acquire appropriate social/interpersonal skills 

• achieve academic competencies 

Other features of the combined PBL and APL strategy 
are as follows: First, the project provides practical value 
to the users who sponsor it. Second, the project provides 
a vehicle for students to experience ‘reaT situations 
(people, building, office etc). Third, they interview the 
real user(s), who are accessible without the need for 
them to do much traveling when the project is on the 
campus. Fourth, students feel that they can contribute to 
a ‘rear situation, thus they are more committed to 
putting in extra efforts in their work. Fifth, students tend 
to deepen teamwork and control the project work better 
to meet the expectation of the users. Sixth, the users are 
involved in attending their presentations and providing 
them with feedback; and seventh, students can better 
understand their strengths and weaknesses, and areas of 
knowledge and skill. They are able to understand the 
necessary skills and knowledge required by the IS 
profession now and in the future. They will also be more 
aware of their development needs and put forth a plan to 
cover the gap between their existing strengths and those 
currently required by the profession. The teaching and 
learning process can be more effective as people outside 
the teaching team are also involved. Indeed, the external 
feedback mechanism from the project sponsor is a 
valuable learning experience through which students can 
gauge their own performance and avoid pitfalls in the 
future. 

DIFFICULTIES OF THE 
COMBINED PBL AND APL STRATEGY 

There are a number of difficulties under the PBL 
strategy that need to be overcome. Students require a 
good number of skills for proper PBL, including 
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problem analysis, information search, teamwork, 
communication etc. The PBL treatment is different from 
the traditional approach of teaching. Students need a 
short period of time to accustom to a new way of 
learning and teaching, especially if the entire curriculum 
does not follow the PBL approach. PBL requires good 
problem cases, which can cover the course objectives 
and address the common contemporary issues. They 
must be interesting and should motivate students to 
learn. APL relies on real-life projects, which enable 
students to learn by doing and learn from their own 
mistakes. 

Many students have the concept of a 50% passing mark 
for a subject. However, an IS project needs to be 
completed with 100% satisfaction of its users. Ensuring 
students are able to conplete a project successfully 
poses some challenges. The question is where to secure 
projects that students are capable of completing, thus 
bringing benefits to the sponsors. The selection of 
projects needs to be based on the following 
considerations: 

• The project addresses the needs of the curriculum and 
is of appropriate standard. 

• The project provides practical value to the user 
department. 

• The project is both interesting and meaningful to 
students. 

• The project is challenging and provides sufficient 
opportunities for students to learn. 

• The project size is reasonable for the project team to 
work on. 

• The degree of difficulty is sufficient for students to 
develop the various skills of the profession. 

• The operation of the department may not be affected 
even if the project is not successful. 

• The users are ready to be involved and to participate 
in the project development process. 

• The users are willing to support the learning and 
teaching process. 



STUDENTS’ PERCEPTION 
OF THE PBL AND APL STRATEGY 

In the Fall of 2001, both authors applied the combined 
PBL and APL strategy to their Systems Analysis and 
Design courses. It has been found that this experience is 
very motivating to students and instructors. The 
instructors were motivated because students gradually 
became active learners as opposed to passive listeners. 
The students were also motivated because they learnt the 
theory and could apply their knowledge to develop 
project, which required them to submit to their clients 
before the deadline. The interactions with students were 
higher as they raised more questions for discussion. To 
gauge students’ perception of our approach, at the end of 
the semester we conducted a survey of the different 
learning issues as compared with traditional teaching 
methods. A total of 34 questions were asked that 
addressed different learning activities during the course. 
Students were asked to respond to each question 
according to the following scale: 1 — Strongly disagree, 
2 — Disagree, 3 — ^Neutral, 4 — ^Agree, and 5 — Strongly 
agree. The survey data, for both the PBL and APL 
approaches to learning and teaching, have been 
statistically evaluated using the Mann-Whitney Test. 
The average score, for the Hong Kong students and the 
US students, is provided for each question. The Mann- 
Whitney Test was used to test the mean differences 
between the two groups with statistical significance at 
the level of 5%. Those results that are marked with 
indicate that the mean score is significantly different. 
When they are marked with this indicates that the 

mean score is barely significant. The results are depicted 
in the following tables, and this is followed by an 
analysis of the responses. 

ANALYSIS OF THE RESULTS 

Regarding PBL, the analysis of the findings provides the 
following results: 

Concerning the appropriateness of PBL for the subject 
of programming: the US students regard PBL as more 
appropriate. The US students give this an average score 
of 42.38 versus the HK students who give this an 
average score of 27.67. When the students are asked 
whether the PBL approach improves your learning style, 
the HK students are more likely to consider that their 
learning style had improved. The HK students give this 
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TABLE 1 
PBL STRATEGY 





HK 


US 










Mean 


Mean 


Mann- 






Measurements 


Score 


Score 


Whitney 


Z 


p - value 


Subject Appropriateness 


It is appropriate for Systems Analysis 


32.88 


31.71 


435.0 


-0.280 


0.779 


It is appropriate for Systems Design 


30.17 


37.26 


351.5 


-1.603 


0.109 


It is appropriate for Programming 


27.67 


42.38 


244.0 


-3.113 


0.002* 


It is appropriate for Systems Development 


31.12 


35.33 


392.0 


-0.954 


0.340 


Skills Development 


It develops your analytical thinking skills 


33.48 


30.50 


409.5 


-0.665 


0.506 


It develops your critical thinking skills 


33.83 


29.79 


394.5 


-0.912 


0.362 


It develops your problem-solving skills 


32.09 


33.33 


434.0 


-0.283 


anil 


It enables you to practice leadership skills 


30.52 


36.55 


366.5 


-1.323 


0.186 


It develops project planning and control skills 


31.64 


34.26 


414.5 


-0.590 


0.555 


It develops life-long learning skills 


31.59 


34.36 


412.5 


-0.591 


0.555 


It develops the various skills for the IT profession 


31.06 


35.45 


389.5 


-0.953 


0.340 


You have more confidence in presentation 


34.26 


28.90 


376.0 


-1.164 


0.244 


You have more confidence in report writing 


35.05 


27.29 


342.0 


-1.707 


0.088 


You can apply knowledge to a problem situation 


34.27 


28.88 


375.5 


-1.260 


0.208 


Perception of Learning Effectiveness 


You like this approach to learning 


34.91 


27.57 


348.0 


-1.598 


0.110 


The approach improves your learning style 


35.30 


26.76 


331.0 


-1.869 


0.062 ** 


You learn information search via the Web 


34.73 


27.93 


355.5 


-1.507 


0.132 


You understand the subject better 


33.42 


30.62 


412.0 


-0.604 


0.546 


You learn more of the subject 


34.23 


28.95 


377.0 


-1.150 


0.250 


You require more independent learning 


35.93 


25.48 


304.0 


-2.214 


0.027 * 


It motivates you to learn better 


36.86 


23.57 


264.0 


-2.802 


0.005 * 


It motivates you to put in more effort 


34.31 


28.79 


373.5 


-1.214 


0.225 


It provides a better learning environment 


33.80 


29.83 


395.5 


-0.873 


0.383 


You know better what you need to learn 


34.29 


28.83 


374.5 


-1.180 


0.238 


It develops your IT professional knowledge 


36.57 


24.17 


276.5 


-2.676 


0.007 * 


Teamwork 


You learn a problem solving process 


33.62 


30.21 


403.5 


-0.733 


0.464 


You require more teamwork 


35.78 


25.79 


310.5 


-2.180 


0.029 * 


You learn more from team members 


35.81 


23.80 


266.0 


-2.556 


0.011 * 


It requires more reading 


33.74 


29.95 


398.0 


-0.810 


0.418 


It raises questions for discussion 


32.73 


32.02 


441.5 


-0.149 


0.881 


Comparison with Traditional Approach 


It should have a quiz or test to assess you 


34.74 


27.90 


355.0 


-1.490 


0.136 


It is better than the traditional approach 


31.01 


35.55 


387.5 


-0.980 


0.327 


Workload is the same as the traditional approach 


33.52 


30.40 


407.5 


-0.677 


0.499 


Overall assessment is the approach is effective 


32.53 


32.43 


450.0 


-0.023 


0.982 
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TABLE 2 
APL STRATEGY 





HK 


US 










Mean 


Mean 


Mann- 






Measurement 


Score 


Score 


Whitney 


Z 


p - value 


Subject Appropriateness 


It is appropriate for Systems Analysis 


32.29 


35.76 


442.5 


-0.764 


0.445 


It is appropriate for Systems Design 


32.80 


34.80 


464.5 


-0.471 


0.637 


It is appropriate for Programming 


37.77 


25.52 


311.0 


-2.580 


0.010* 


It is appropriate for Systems Development 


35.93 


28.96 


390.0 


-1.682 


0.093 


Skills Development 


It develops your analytical thinking skills 


32.87 


34.67 


467.5 


-0.406 


0.685 


It develops your critical thinking skills 


30.99 


38.20 


386.5 


-1.616 


0.106 


It develops your problem-solving skills 


32.90 


34.63 


468.5 


-0.388 


0.698 


It enables you to practice leadership skills 


31.28 


37.65 


399.0 


-1.413 


0.158 


It develops project planning and control skills 


33.13 


34.20 


478.5 


-0.242 


0.809 


It develops life-long learning skills 


34.15 


32.28 


466.5 


-0.404 


0.686 


It develops the various skills for the IT profession 


34.79 


31.09 


439.0 


-0.843 


0.399 


You have more confidence in presentation 


36.71 


27.50 


356.5 


-1.996 


0.046* 


You have more confidence in report writing 


36.94 


27.07 


346.5 


-2.121 


0.034* 


You can apply knowledge to a problem situation 


35.99 


28.85 


387.5 


-1.614 


0.106 


Perception of Learning Effectiveness 


You like this approach to learning 


34.21 


32.17 


464.0 


-0.445 


0.656 


The approach improves your learning style 


34.51 


31.61 


451.0 


-0.630 


0.529 


You learn information search via the Web 


34.63 


31.39 


446.0 


-0.707 


0.479 


You understand the subject better 


34.31 


30.43 


416.5 


-0.865 


0.387 


You learn more of the subject 


33.90 


32.76 


477.5 


-0.241 


0.809 


You require more independent learning 


37.65 


25.74 


316.0 


-2.531 


0.011* 


It motivates you to learn better 


39.60 


22.09 


232.0 


-3.707 


0.000* 


It motivates you to put in more effort 


32.20 


35.93 


438.5 


-0.826 


0.409 


It provides a better learning environment 


31.83 


36.63 


422.5 


-1.042 


0.297 


You know better what you need to learn 


35.56 


29.65 


406.0 


-1.279 


0.201 


It develops your IT professional knowledge 


36.95 


27.04 


346.0 


-2.110 


0.035* 


Teamwork 


You learn a problem solving process 


36.98 


27.00 


345.0 


-2.242 


0.025* 


You require more teamwork 


33.71 


33.11 


485.5 


-0.132 


0.895 


You learn more from team members 


34.58 


28.24 


362.0 


-1.353 


0.176 


It requires more reading 


35.20 


30.33 


421.5 


-1.065 


0.287 


It raises questions for discussion 


33.20 


34.07 


481.5 


-0.186 


0.853 


Comparison with Traditional Approach 


It should have a quiz or test to assess you 


35.59 


29.59 


404.5 


-1.294 


0.196 


It is better than the traditional approach 


32.33 


35.70 


444.0 


-0.723 


0.470 


Workload is the same as the traditional approach 


36.69 


27.54 


357.5 


-1.969 


0.049* 


Overall assessment is the approach is effective 


33.94 


32.67 


475.5 


-0.280 


0.779 
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an average score of 35.30 versus the US students who 
give this an average score of 26.76. 

In addition, the HK students consider that they can 
acquire greater benefit from PBL, in the following areas, 
than the US students: 

• You require more independent learning 

• It motivates you to learn better 

• It develops your IT professional knowledge 

• You require more teamwork 

• You learn more from team members 

Regarding APL, the analysis of the findings is presented 
as follows: 

Concerning the appropriateness of APL for the subject 
of programming: the HK students regard APL as more 
appropriate for programming. The HK students give this 
an average score of 37.77 versus the US students who 
give this an average score of 25.52. 

In addition, the HK students consider that they can 
acquire greater benefit from APL, in the following areas, 
than the US students: 

• You have more confidence in presentation 

• You have more confidence in report writing 

• You require more independent learning 

• It motivates you to learn better 

• It develops your IT professional knowledge 

• You learn a problem solving process 

• You require more teamwork 

Finally, the HK students regard the workload as much 
higher in APL when it is compared with the traditional 
approach. 

Nevertheless, our observations constitute an isolated 
study of just two universities. More formal studies are 
required to draw a general conclusion. However, our 
study proves that students are far more professionally 



active under the combined PBL and APL strategy than 
under traditional teaching. Some areas in whidi they are 
professionally active include commitment to teamwork 
and taking up team-member responsibilities, creative 
thinking, analytical skills, communication skills, and 
leadership skills. 

STUDENTS’ COMMENTS ON PBL AND APL 

Both the Hong Kong and US students made comments 
on the two approaches and the following are some of the 
comments: 

PBL 

• Hong Kong students. “Problem-based Learning 
provides (an) easier way to learn the subject. It helps 
(us) to understand the subject more clearly.” 

• “PBL provides more cases to think and less need to 
memorize.” 

• “Problem-based learning requires more group work, 
but overall this is an innovative and effective teaching 
method.” 

• US students. “The teamwork aspect of the PBL was 
a great idea because you learned to work together and 
have actual hands-on effect. It was something that you 
might be assigned in your future job.” 

• “The PBL helped me to learn better but for course 
purposes projects should be limited to possibly one or 
to (two) phases of a system.” 

• “Problem-based learning makesme feel I am applying 
my knowledge to solve real problems.” 

APL 

• Hong Kong students. “I prefer APL but the workload 
is heavy and time is not sufficient to finish the 
project.” 

• “For the Action Project Learning, it requires more 
tutors to support the team. It will make the project 
more effective.” 

• “I think Action Project Learning is better than 
problem-based learning, because actual work is done 
in action project(s) while problem-based only requires 
the submission of reports.” 
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• “Action Project sets a goal for my learning and leaves 
room for me to decide how much I would like to 
learn.” 

• US students. “Our team was very enthusiastic to do 
our project as a group. Without application of APL I 
would not have learned that much.” 

• “The project selection was great because I felt like I 
was working for that company plus when we have 
presentations they are more interesting.” 

• “I learned the material better by actually using the 
information I learned inmy project. Therefore Action 
Project Learning is an excellent approach.” 

• “In general I found it a very effective way to learn the 
course material and I think it will aid in my 
professional IT career.” 

CONCLUSION 

The combined PBL and APL strategy, which we have 
applied to our courses, proved to be successful in many 
ways. Based on our experiences and the students’ 
responses, we believe that the PBL strategy has helped 
our students to learn the necessary knowledge and skills, 
and that it has improved their motivation. This has also 
been demonstrated in the overall outcome of the courses 
in both institutions. The students’ average grades were 
higher than those on the same course when a traditional 
teaching method was applied. The improvements in their 
motivation have been demonstrated in their interaction 
with other students and with the instructors. We 
measured their other course-related activities (beyond 
attending classes) and found that they have learned 
problem analysis, planning, teamwork skills and how to 
do research. In addition, the APL strategy helps students 
to become actively involved in solving real-life 
problems. This task involves skills inhuman interaction 
that are not attainable in a traditional classroom-teaching 
method. It can only be accomplished through inter- 
actions with project sponsors and amongst teammates. 
The instructors have evaluated the amount of interaction 
by gathering data. On average, students met forone hour 
each week to discuss their project related issues. 

Overall, the PBL and APL learning strategy yielded 
several other positive results that are worth noting. The 
rate at which students learn through educational 
activities (such as reading, searching and 
communicating) is much higher when compared to 



traditional teaching. This remark is echoed in the 
responses of the students from both schools in their 
survey ratings and in their written comments. We must 
also mention that this learning strategy requires stringent 
effort both on the part of the students and on the part of 
the instructors. However, the learning outcome is so 
beneficial that it is worth the extra effort. It has also 
been found that peer learning and the concept of a 
deadline drives students to work towards their goals. 
Combining the PBL and APL strategies requires that 
instructors write good problem cases and find suitable 
real-life projects. A balance in the use of teaching time 
needs to be made, fcr PBL and APL, to maximize the 
teaching and learning effectiveness. 

REFERENCES 

Biggs, J. (1999). Teaching for Quality Learning at 
University, Open University Press, 1999. 

Boud, D. and Feletti, G. I. (1997). The challenge of 
problem-based learning, London: Kogan Page, 
1997. 

Daqing, G and Bertin, V., (2001), Teaching Design 
Integration through a Hands-on Approach, Further 
Case Studies of Improving Teaching and Learning 
from Action Learning Project, ISBN 962-237-295- 
0,2001, pp. 3-12, 2001. 

Horwood, J. and Miliszewska, I. (2001). Towards 
Lifelong Learning: Evolution through Technology, 
Enhancement of Quality Learning Throu^ Infor- 
mation and Communication Technology, Vol 1, 
2001, pp. 593-500, 2001. 

Johnson, C. (1998). The Essential Principles of Action 
Learning. Journal of workplace learning, (10) 6, pp. 
12-23, 1998. 

Kember, D. (2001), The Outcomes of the Action 
Learning Project, FurtherCase Studies of Improving 
Teaching and Learning from Action Learning 
Project, ISBN 962-237-295-0, 2001, pp. 421-434, 
2001. 

Kazlauskas, E. J. (2001), Integrating University-based 
Technology Projects into Education Programs, 
Enhancement of Quality Learning Through Infor- 
mation and Communication Technology (ICT), 
Proceedings of ICCE/SchoolNet2001, Vol 1, pp. 
269-275, 2001. 



Proceedings of the 17^ Annual Conference of the International Academy for Information Management 



275 



o 

ERJC 



284 



Shi Yi-zheng and Siu Wai-sum, (2001), Learning by 
Doing through ‘In-firm’ projects , Further Case 
Studies of Improving Teaching and Learning from 
Action Learning Project, ISBN 962-237-295-0, 
2001,pp. 3-12, 200L 



Walker, M. (2000). Learning how to Learn in a 
Technology Course: A Case Study, Open Learning, 
Vol 15, No. 2, pp. 173-189,2000. 



276 



Proceedings of the 17^ Annual Conference of the International Academy for Information Management 



INFORMATION SYSTEMS DEVELOPMENT 
USING CROSS-DISCIPLINARY TEAMS 



Kathleen Wright 
Salisbury University 

Karen Papke-Shields 
Salisbury University 

Mary J. Granger 
George Washington University 



ABSTRACT 

Increasingly, businesses are using cross-functional teams to support system development projects, creating a need 
to establish a common understanding of the systems development process. This paper describes a cross-disciplinary 
systems project designed to simulate the business team experience by grouping operations management students with 
information systems students to develop a forecasting application. Results of a pilot implementation are discussed. 



INTRODUCTION 

Because of the emphasis on team interaction and group 
processes in business, many institutions in business 
education have incorporated group projects into their 
course curricula. Typically, group projects are assigned 
to teams within a single course setting. This has several 
practical advantages, most notably that the students have 
a common understanding of how to complete the tasks 
based on the instruction they have received in the class. 
And, although the students may have different skill 
levels and varying interest in achievement, their 
objectives in terms of mastering the course content are 
aligned. 

The single course setting, however, has limitations as a 
surrogate for the business environment, which is moving 
towards the use of cross-functional teams as offering 
many potential benefits. For example, new product 
development and process reengineering projects often 
involve individuals across operations, engineering, 
marketing and finance. In applying information 
technology to a business problem, team diversity is 



generally the norm, with the primary interaction 
occurring between the end-user as the content expert and 
the information systems specialist as the facilitator 
(Kirsch, et.al., 2001; Corritor and Gasper, 2001). 

Dynamics between end-users and information systems 
specialists have also been evolving. Increasingly, 
managers recognize the strategic importance of 
information systems to the organization and are 
encouraging user participation in development projects 
(Hartwick and Barki, 1994; Sambamurthy and Zmud, 
1996). The advent of fourth-generation programming 
tools and the application prototyping approach to 
systems development have enabled end-users to take a 
significantly more active role. Rather than simply 
articulating the business needs at the start of a project 
and waiting for the finished system, users are often 
involved throughout the development life cycle, creating 
logical data views, assisting in the interface design and 
testing data outputs. This growth in user involvement 
appears to indicate a need for end-users to gain more 
exposure to systems development projects, increasing 
their understanding of the process. 
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The purpose of the Forecasting Project is to more 
closely simulate the development process in a controlled 
environment, using student teams from across 
disciplines. To complete the application, undergraduate 
students in an Operatiais Management (OM) course are 
teamed with (IS) students. The objectives of the 
assignment are to: 

• Reinforce course content. The OM students apply 
their knowledge of forecasting models. The IS 
students gain experience in applying programming 
techniques to mathematical formulae. 

• Experience the roles of end-user and developer. 

Not only are the OM students content providers, they 
also are responsible for communicating the fore- 
casting model requiremaits clearly and for providing 
timely feedback during the testing phase. The IS 
students must, in turn, develop their listening skills to 
understand the requirements and translate them into 
the constraints of the development tool. 

• Develop an understanding of the relationship of 

application development to testing. Because the 
application must provide accurate forecasts, the IS 
students must not simply provide a solution, but must 
also work with the OM students to determine its 
accuracy. This subtly alters the emphasis of the 
project by creating two distinct phases: initial 

development and testing. 

• Learn to manage a project process across 
heterogeneous teams. Successful completion 
requires transfers of information and materials 
between students with varying backgrounds, time 
constraints and responsibilities. This adds 
considerable complexity to the project, requiring 
students to communicate continually to move the 
project towards a solution. 

RESEARCH QUESTIONS 

Given the relative novelty of this approach to 
applications development and student teams (Corritor 
and Gasper, 2001), the research questions are broad and 
exploratory. The first question is simply whether the 
Forecasting Project is feasible, given that it requires 
careful coordination across two courses? Second, does 
the project achieve the stated objectives? A corollary 
question is what interventions can the instructors use to 
ensure that these objectives are achieved? Finally, do 
students find projects of this type to be valuable? 



IMPLEMENTATION METHODOLOGY 

The Forecasting Project is introduced to the OM and IS 
students in the second half of their semester course. At 
this point, the OM students have mastered six 
forecasting models: naive, moving average, weighted 
moving average, exponential smoothing, trend analysis 
and trend with seasonality. The IS students have 
covered basic programming techniques, including 
decision and repetition structures, using Visual Basic 6.0 
as the software development tool. 

A user graphical interface template has been developed 
to provide the IS students with a starting point, and to 
enable them to focus solely on the program code 
required to implement the various forecasting models. 
The input interface (Figure I) uses controls that the IS 
students are already familiar with for storing data values 
and for obtaining additional user information that is 
specific to each model. A second preprogrammed form 
has also been developed to enable students to see the 
data as it appears when graphed. 

The two simplest forecasting models, naive and moving 
average, have been programmed into the application 
template to enable students to see a sample output 
(Figure 2) and to demonstrate common repetition 
algorithms for looping through control arrays. 
Similarly, standard output tasks, such as a graph 
showing the projected forecasts versus the actual data, 
have already been completed. The template provides 
the IS students with an initial prototype to show to the 
OM student. In this respect, the Forecasting Project 
simulates modification of an existing application to meet 
user requirements rather than creation of a new 
application from scratch. 

Teams meet initially to exchange infcxmation, discuss 
the remaining forecasting models and to draft a problem 
statement, set of specifications and a timeline for initial 
development and testing. The latter serves as a contract 
and is a critical component of the project, as the OM 
students are responsible for assuring the accuracy of the 
final product and must actively participate in application 
testing. The resulting documentation is submitted to 
both instructors as the first deliverable. 

Subsequent discussions can take place either in person 
or through email as the IS students deliver the initial 
version of the completed application to their OM 
counterpart. A copy of the completed application is also 
provided to the instructor for additional quality control. 
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FIGURE 2 

THREE MONTH WEIGHTED AVERAGE 
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The OM students are responsible for testing the 
application and determining whether the output is 
accurate, documenting errors and providing feedback to 
the IS students for correction. As constant communica- 
tion with each other and with the instructors has been 
found to be a critical component of the success of 
student teams (Stephens and Myers, 2001), 
documentation created through this jrocess is also sent 
to the instructors, who can intervene if an impasse 
occurs. 



PILOT TEST- 

IMPLEMENTATION AND RESULTS 

A pilot test of the Forecasting Project was conducted 
during the spring semester of 2002. Twenty-eight 
undergraduate IS students were teamed with forty-five 
OM students. Because of the timing of the project in 
relation to the spring semester break, only two 
forecasting models, weighted moving average and 
exponential smoothing, were required. In addition. 
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although the project assignment and prototype template 
was presented to both IS and OM students in a joint 
session, the IS students worked fairly independently. 
The problem statement and specifications were drafted 
solely by the IS students. The joint testing and 
correction phase was also eliminated because of time 
constraints. Completed applications were given directly 
to the instructor, who graded the results and made 
corrections prior to returning the projects. The IS 
students were then responsible for delivering the 
corrected application to the OM students. The OM 
students then used the application to complete an 
assignment about selecting the best quantitative 
forecasting model. 

Although some of the critical components of the 
exercise were missing, the pilot did result in some 
interesting findings. For example, the OM students 
learned the forecasting formulas for a single period and 
then applied the same formula to subsequent periods. 
While the IS students were able to follow the formula 
logic, they were initially confused by idea of forecasting 
over multiple periods and had some difficulty in 
incorporating the formula logic within a repetition 
structure. This clearly demonstrated some of the 
difficulties in translating user requirements into the 
logic required by the software development tool. As one 
IS student stated: 

I think I am having a little difficulty because 
I’m not at all familiar with the formulas or 
forecasting. My OM counterparts explained the 
equations well and how to use them; I was just 
slightly confused on what the equations would 
give you. I think this project is really helpful in 
that you realize that each side needs to clearly 
explain their part of the project and ask any and 
all questions (that you) have. 

After initially meeting with their OM counterparts, the 
IS students spent class time developingcode algorithms. 
This provided an opportunity for the IS instructor to 
reinforce basic programming concepts (e.g., using 
variables to store information as calculations are rolled 
forward in time.) Working through the algorithms in 
class, however, created a second problem that was not 
revealed until the completed applications were 
submitted to the IS instructor. In several of them, it was 
apparent that the IS students had simply followed the 
algorithm logic and, having obtained output, stopped 



without verifying the validity of their results. Typically 
the coding errors were subtle, pointing to the wrong data 
value (e.g., the current period instead of the prior) in 
exponential smoothing, or assigning the weights in re- 
verse order for the weighted average forecasting model. 

Because of time constraints, these errors were simply 
corrected by the IS instructor before they were given to 
the OM students. This eliminated the testing phase, and 
by extension, a significant opportunity for OM and IS 
students to work together to identify and correct errors. 
The result was grousing on both sides. From the pers- 
pective of the OM student, the lack of participation and 
project ownership resulted in perceptions that the bene- 
fits of the experience were solely accruing to the IS 
students: 

I thought it was interesting, but probably more 
from the perspective of the IS student, because 
I am an IS major and did not get to do this 
project from the IS side. 

The project really didn’t teach me anything 
more than what I could have learned from 
studying. I understood the models, but having 
to collaborate with the IS student was somewhat 
of a hassle. Having the IS student create the 
application didn’t teach us anything. 

It seemed the group member from the other 
class had to do most of the work. All we 
really did was explain the formulas to him. 

The IS student comments revealed that they perceived 
themselves as doing all the work. 

It seems that we are doing twice as much work 
as the OM class. Not only do we have to write 
the program, but we have to learn their part of 
the project as well. 

I think meeting with the OM student was kinda 
pointless because all we needed was the 
formulas that could have been given by the 
professor! I 

Nevertheless, the majority of the student comments were 
positive. The following two statements, from an OM and 
IS student, respectively, are particularly articulate 
representations: 
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I thought the project was very helpful. 
Although I already had a strong understanding 
of the fore-casting models, it was beneficial to 
use the program and interact with others in 
order to find a feasible solution. 

I think it’s great for “real world” experience. I 
am relieved that the interface and most of the 
code is already in place so I can focus entirely 
on coding just the two required models. It’s 
interesting how each student must trust one 
another to achieve desired results. 

Although the results are preliminary, the responses to 
surveys administered independently to the OM and IS 
students (Appendices A and B) also indicated that the 
students perceived the project to be beneficial. The IS 
students indicated moderate agreement with the 
statement that “The benefits of the assignment are clear 
to me.” Likewise, the OM students showed moderate 
agreement with 

Overall, I believe this project was a good 
example of the “real world” because of the 
interaction required between individuals with 
different “expertise.” 

It is interesting to note, however, that while the OM 
students appeared to be confident in their explanations 
of how the forecasting models worked, their IS 
counterparts were not equally convinced that the OM 
students clearly understood the models. 

DISCUSSION AND CONCLUSION 

Given that this was the first implementation of the 
Forecast Project and the timing constrained both the 
interactions with OM/IS students and the testing phase, 
the results were encouraging. For the most part, the 
students enjoyed the interaction with others outside of 
their class although tension was initially high until 
students began to work with each other and realized that 
the project was “doable.” The IS students clearly 
benefited by exercising their listening skills and using 
information, not pre-packaged by the instructor, to craft 
a logical solution. The benefits were not as obvious to 
the OM students (after all, how oftoi do end-users 
readily embrace information systems change when there 
is already a working solution?) Nevertheless, many did 
recognize the value of having to explain the various 



models to their IS counterparts. It will be very 
interesting to see if increasing the joint OM/IS responsi- 
bilities in a future implementation positively affect OM 
student perceptions of project involvement and owner- 
ship. 

With regard to project feasibility, the project appears to 
be feasible, at least as it was constructed for the pilot 
test. Requiring testing as a project phase will add 
another dimension and challenge to the OM/IS student 
dynamics. The students are not the only ones affected, 
however. The astute student comment regarding trust in 
the student teams also applies to the interaction of the 
instructors. Guiding students through any project pro- 
cess requires considerable effort, but in this case the 
potential difficulties are compounded by the involve- 
ment from students from other classes. Coordinating 
with a second instructor in another discipline requires 
one to enlarge his or her own perspective to encompass 
the objectives ofboth. In this respect, attempting inter- 
disciplinary projects also creates a unique opportunity to 
the instructors for learning and professional growth. 

If successful, the implementation of the Forecasting 
Project will present additional opportunities for research 
and further study. For example, a segmentation of the 
pilot test IS student data by gender indicated that female 
students valued the social interaction of the pilot more 
highly than the male students who were more inclined to 
work independently. Additional experience with 
projects like the Forecasting application may help to 
provide insights as to interventions that improve the 
process of application development for both groups. 

Finally, the processes and methodologies that make this 
particular project successful can be adapted to simulated 
development projects involving other business 
disciplines, such as marketing, accounting and finance. 
Because “real world” systems develq^ment projects are 
requiring more group collaboration, the “soft” skills 
needed for effective communication are gaining in 
importance. A simulated development project, such as 
the Forecasting application benefits all participants by 
requiring them to be interdependent, to listen carefully, 
and to speak precisely to arrive at a solution. While the 
project benefits in terms of reinforcing programming 
technique are obvious, business students from other 
disciplines also gain from the exposure to the 
development process and learning how to take a more 
active role. 
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APPENDIX A 

RESULTS OF THE OM STUDENT SURVEY 



Survey Question (5 point Likert-type scale) 


Mean 


Mode 


Std. Dev. 


At the beginning of the project I felt comfortable giving instructions to my IS 
counterpart. 


4.20 


5 


0.91 


I understood the formulas for the different forecasting models so that I could 
explain them to my counterpart. 


4.63 


5 


0.77 


Explaining the formulas to my IS counterpart helped me to better understand the 
different forecasting models. 


3.98 


4 


1.12 


Explaining the project to my IS counterpart helped me to better understand how a 
quantitative model is selected. 


4.13 


4 


0.97 


I found my IS counterpart easy to work with. 


3.93 


5 


1.12 


I was able to meet with my IS counterpart as needed 


3.45 


3 


1.15 


I received updates from my IS counterpart about where s/he was with in her/his 
work. 


3.03 


3 


1.39 


My IS counterpart contacted me if they needed further clarification or explanation. 


3.23 


5 


1.44 


Overall, I believe this project helped me better understand forecasting. 


3.85 


4 


0.98 


Overall, I believe this project was a good example of the ‘Veal world” because of 
the interaction required between individuals with different “expertise.” 


4.03 


4 


0.86 
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APPENDIX B 

RESULTS OF THE IS STUDENT SURVEY 



Survey Question (7 point Likert-type scale) 


Mean 


Mode 


Std. Dev. 


I found the instructions I received from my OM counterpart to be confusing. 


4.11 


5 


1.50 


It is difficult to take the formulas as written and translate them into code 
statements. 


4.39 


5 


1.89 


My OM counterpart did not appear to understand how to perform the forecasting 
calculations.* 


2.15 


1 


1.38 


Meeting face to face to go over the forecasting instructions was very helpful. 


5.22 


5 


1.50 


I went back to my OM counterpart several times to clarify my understanding of 
how the formulas worked.* 


3.00 


1 


2.06 


I find it easier to take the written documentation and work through the logic on my 
own. 


4.63 


6 


1.69 


Writing the code took a lot of my mental effort. 


4.75 


5 


1.69 


The benefits of the assignment are clear to me. 


5.54 


6 


1.17 


I found it difficult to explain to my counterpart how the program could achieve the 
expected result. 


4.04 


5 


1.56 


Writing psuedocode helps me to think through the programming steps more clearly. 


5.00 


6 


1.44 



*Reverse coded 
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ABSTRACT 

As on-line education becomes more pervasive and increasingly acceptable in society, there is a need to critically 
examine the merits and underlying assumptions driving the justification, design, and teaching of such courses. In 
this paper, we take a first step in this direction by uncovering several symbolisms, in the form of myths and 
metaphors, discernable in the discourse on online learning. Future research will involve an empirical examination 
regarding the validity and consequences of these myths/metaphors identified, specifically in the context of 
information systems education. 



INTRODUCTION 

According to estimates from IDC, a Framingham, 
Massachusetts-based market research firm, the 
worldwide e-leaming market will grow from $2.2 billion 
in 2000 to $18.5 billion by 2005 (Moore, 2001). Peter 
Drucker, economist and futurist, caused quite a stir in 
the academic community when, in an interview for 
Forbes magazine, he stated that universities thirty years 
from now would be “relics ... [and] won’t survive” 
(Lenzer and Johnson, 1997, p. 127). Money Magazine 
has said, “Online education may be the greatest 
technological advance to hit academia since the No. 2 
pencil” (Clarke, 1998, p. 66) and John Chambers, the 
chief executive officer of Cisco Systems, has declared 
that “e-leaming is the next major killer app”of the 
Internet (“The Virtual Classroom,” 2000; Moore, 2001). 
In fact, it has become such big business, that major 
investors, such as Michael Milken’s Knowledge 
Universe and Paul Allen’s click21earn.com, are jumping 
on the bandwagai to offer courses to both college 
students and corporate trainees (Molenda and Harris, 
2001). Media financier, Herbert Allen, Jr., is the prime 
backer of Global Education Network (GEN), a 



consortium of top-notch liberal arts colleges and ivy- 
league universities, including Brown, Wellesley and 
Williams, brought together to provide online education 
(“The Virtual Classroom,” 2000; Weber, 2000). 
Additionally, some universities, such as University of 
Maryland and Columbia University, are forming their 
own for-profit subsidiaries to offer online degree 
programs in order to exploit their “intellectual capital, ” 
presumably without diluting the reputation of their 
brick-and-mortar university (Molenda and Harris, 200 1). 
Thus, it is evident that higher education institutions, 
including those with outstanding “bridc-and-mortar” 
historical reputations, as well as major investors, are 
charging ahead in an effort to capture a share of the 
online education market. This pattern is noticeable 
among Business Schools in general, which are 
increasingly leaning toward offering online programs, 
especially in the area of Information Systems whose 
courses are in high demand. But what are the driving 
forces behind this trend toward online education? 

A number of reasons are cited in the literature. Adult 
students, those 25 years of age and over, currently 
represent nearly one-half of credit students enrolled in 
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higher education (Kasworm, Sandmann, and Sissel, 
2000), yet they have greater work and family 
responsibilities that prevent them from attending school 
full time. These students, then, may be choosing an 
online course because of lifestyle issues. Another 
driving force is the increasing cost of education and the 
lack of financial aid available for those students who 
would like to attend full time (Raymond III, 2000). 
Thus, students may be forced to choose an online course 
for financial reasons. The pervasiveness of information 
technology into the workplace has also created a 
growing societal demand for continuing education, 
requiring many workers to acquire new skills (Molenda 
and Harris, 2001). In this day and age most large 
companies are sold on the idea that continual training 
and retraining of employees will pay off big (“The 
Virtual Classroom,” 2000). In addition, technology in 
and of itself is a major driving force. As more and more 
people have access to a computer and the Internet, it is 
easy to fall into a mindset of thinking “since it’s there, 
we should use it.” Finally, as government support for 
public institutions is shrinking, nonprofits are being 
forced to find their niche in the marketplace and operate 
more like a for-profit business. Many of these 
institutions are turning to online education as a way to 
support themselves within the marketplace (Molenda 
and Harris, 2000). As such, for-profit and non-profit 
institutions are competing for the upper hand in the 
headlong rush to implement new technology. Herbat 
Allen, Jr.’s for-profit venture, GEN, has its own 
conception of the driving force behind this trend and is 
targeting four distinct markets with its online courses: 
“lifelong learners” who graduated from college decades 
ago but want to challenge themselves; college students 
at smaller institutions who want to supplement their 
studies with courses not offered on their real-life 
campuses; high-school students gearing up forcompeti- 
tive college-admission processes; and students overseas 
who want access to U.S. institutions (Weber, 2000). 

While online education is being touted aggressively, 
there are some opposing voices that seem to be muted 
by the sheer weight of tedinological and economic 
determinism, the basis for much of the uncritical 
optimism regarding on-line teaching and learning. 
Among those voices are 840 faculty members from the 
University of Washington (UW) who signed an open 
letter objecting to Governor Gary Lx>cke’ s 1 998 proposal 
of a distance-learning initiative that suggested 
traditional state-university courses could be partly 
replaced by online learning (Cleary, 2001). These 



opposing faculty argued that education should not be 
reduced to “the downloading of information, much less 
to the passive and solitary activity of staring at a screen” 
(Cleary, 2001). It is worth noting that the UW 
professors are not alone in their skepticism of online 
education. When Peter Lange, the Provost of Duke 
University, was approached by GEN he was impressed 
by what GEN had to offer. Yet, he expressed his 
reservation regarding the extent to which online 
education can be universally effective, and suggested 
that online learning be approached on a course-by- 
course basis. “Can a company like GEN put together a 
full curriculum which would substitute for what you 
would get if you came to Duke? The answer is clearly 
no,” he says (Weber, 2000). William S. Reed, the vice 
president for finance and administration at Wellesley, 
declared that he is not sure if Wellesley’s courses can be 
successfully translated into Web pages (Weber, 2000), 
yet Wellesley has agreed in principle to join the GEN 
venture. Even the U.K.’s Open University, an institute 
that creates and studies ways to apply new technologies 
to learning, has some harsh words regarding online 
education. “For online education to become mainstream 
is kind of a depressing thought, because it is such a 
crappy experience. The bottom line is that learning 
online is a soul-destroying experience. It really, really 
stinks. It’s always second best (to face-to-face 
learning)”, said Marc Eisenstadt, chief scientist for the 
Knowledge Media Institute at the U.K.’s Open 
University (Hamilton, 2001). Moreover, thejury is still 
out on the issue of online education at organizations 
such as the AACSB, the primary accrediting 
organization for business schools, which typically house 
Information Systems departments as well. While the St. 
Louis, Mo., organization “encourages innovation and 
experimentation in education” in its guidelines, the 
AACSB also cautions “simply adopting new 
technologies without thinking about the implications for 
quality assurance would raise troubling questions” 
(Cleary, 2001). The sentiments of those who are erring 
on the cautious side before jumping feet first into online 
education can be properly summed up by Arthur Levine, 
president of Teachers College at ColumbiaUniversity in 
New York, when he said, “We still don’t know if online 
education is any good” (Cleaiy, 2001). 

As IS educators, the authors’ own position in this debate 
is one of cautious optimism — while we see significant 
gains that can be derived from online education, we 
consider the takenfor-grantedness of the presumed 
merits, largely fuelled by financial and technical 



Proceedings of the / 7 * Annual Conference of the International Academy for Information Management 



299 



274 



BEST COPYAVAIUBLE 



opportunism, as being detrimental to the long-term 
interests of the society to produce reflective Information 
Systems practitioners. 

Given the hype that has accompanied this new form of 
learning, and the significant investment of both money 
and time that has gone into creating e-leaming ventures, 
we believe that there needs to be a critical examination 
of the assumptions underlying the discourse on this 
topic, with the goal of separating hype from reality. In 
order to delve deeper into the question of whether online 
education is an appropriate supplement to, or 
replacement for, traditional education, following the 
approach of Hirschheim and Newman (1991), we 
explore some of the symbolisms, namely myths and 
metaphors, pertaining to online higher education, with 
the ultimate goal of investigating if and how they apply 
specifically to information systems education. 

SYMBOLISMS— MYTHS AND METAPHORS 

Symbols embody a community’s views (or theories) 
about particular phenomena, which in turn explain 
behaviors exhibited by members of the community. 
Thus, symbolisms, such as myths and metaphors, can be 
used to make sense of situations that are new, 
problematic, ambiguous, or unsettled (Frost and 
Morgan, 1983). Symbolism revolves around shared 
meanings — patterns of beliefs, rituals and myths, which 
evolve through time and function as social glue, binding 
communities together (Smirdch, 1983). In the realm of 
online education, where there exists for many an 
absence of experience to guide practice, the images and 
ideas embedded in the discourse reflect and 
simultaneously shape people’s views (Schultze and 
Orlikowski, 2001) towards this new form of education. 
Thus, unlike mainstream positivist research studies, 
which are concerned with the empirical testability of 
theories in an attempt to increase predictive 
understanding of phenomena, the current study calls for 
an interpretivist approach, as we are attempting to 
understand phenomena through accessing the meanings 
that participants assign to them (Orlikowski and 
Baroudi, 1991). The interpretive perspective empha- 
sizes the importance of subjective meanings and social- 
political, as well as symbolic, action in the processes 
through which humans construct and reconstruct reality 
(Morgan, 1983, p. 396) and asserts that the language 
humans use to describe social practices constitutes those 
practices (Orlikowski and Baroudi, 1991). According to 
this perspective, an examination of embedded 
symbolisms, or more specifically, myths and metaphors. 



in the language of writers/speakers on online education 
will enable an understanding of the common attitudes 
and beliefs surrounding online education, and thus a 
comprehension of the (socially constructed) reality 
regarding this phenomenon. 

Myth can be defined as 

A dramatic narrative of imagined events, 
usually used to explain origins or transforma- 
tions of something. [It also reflects] an 
unquestioned belief about the practical benefits 
or certain techniques and behaviors that is not 
supported by demonstrated facts (Trice and 
Beyer, 1984, p. 655). 

Myths are often communicated through the telling of a 
story; nonetheless, they are not merely recountings of 
any particular tale, but ways of classifying and 
organizing reality (Polkinghome, 1988, p. 83). Myths 
are devices of mind that have been used throughout time 
to provide explanations, reconcile contradictions, and 
help resolve dilemmas; however, myths have also been 
known to distort images and misdirect attention 
(Bolman and Deal, 1984). Negative traits notwith- 
standing, “myths are necessary to create meaning, 
solidarity and certainty” (Bolman and Deal, 1984) and 
serve the following functions: 1) myths explain, 2) 
myths express, 3) myths maintain solidarity and 
cohesion, 4) myths legitimize, 5) myths communicate 
unconscious wishes and caiflicts, 6) myths mediate 
contradictions, and 7) myths provide narrative to anchor 
the present to the past (Cohen, 1969). An example of 
this is the “myth of academe,” which represents the 
belief that faculty lead lives devoted to the selfless 
pursuit of knowledge in institutions carefully organized 
to support that pursuit (Shaw, 2000). It is important to 
note here that myth is not necessarily empirically 
invalid. Rather, it is a belief that is assumed to be valid 
in the presence of contrary evidence or in the absence of 
any evidence at all. 

Metaphors have been described as a way of under- 
standing and experiencing one thing in terms of another. 
Aristotle spoke of the value of metaphor almost 2,200 
years ago: “Ordinary words convey only what we know 
already; it is from metaphor that we can best get hold of 
something fresh” (Embler, 1966, p. 12). According to 
Morgan (1986, p. 12), metaphors are used as “a way of 
thinking and a way of seeing that pervade how we 
understand our world generally.” A metaphor has the 
power to shape reality and structure the thoughts of the 
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people who are caught up in a particular metaphor and 
itsentailments (Duncan, 1968;Graber, 1976;Lakoffand 
Johnson, 1980). The true effectiveness of metaphors is 
their almost paradoxical ability to point out 
dissimilarities and contrasts between two objects ^\^lile 
simultaneously demonstrating that there are 
considerable similarities between the objects being 
compared (Weaver, 1967). Based on the work of Koch 
and Deetz (1980) we can make the following 
assumptions with regards to metaphors: 1) they are 
fundamental in the Englishlanguage, 2)basic metaphors 
entail others, 3) complex networks of metaphors 
permeate our language, leading us to conceptualize 
certain things in certain ways, 4) metaphors are usually 
coherent and can be traced back to fundamental forms of 
experience, and 5) inconsistent metaphors can be 
explained by tracing them to different realms of 
experience. An example of a metaphor is the phrase 
“time is money”, which entails the idea that time is 
valuable, can be spent, saved, lost, and so on. From this 
example, it is evident to see how metaphors can be 
helpful as carriers of meaning; however, they can also 
be dangerous fantasies “and not suited for guiding 
serious meaning’ (Boland, 1987, p. 367). 

METHODOLOGY 

The following steps outline how went about unearthing 
the myths and metaphors embeddedwithin the literature 
on online education. The first step in discovering the 
myths and metaphors utilized to describe online 
education involved the selection of articles for analysis. 
We used Pro-Quest Direct and ERJC to search for peer- 
reviewed articles that contained at least one of the 
following terms: online learning, online courses, online 
education, online instruction, web-based instruction, 
web-based courses, Internet courses, e-leaming, 
computer-based instruction. Published research articles 
were used as the unit of analysis as they “provide a clear 
sampling frame, as well as the best view of what is 
accepted in the research community” (Watson- 
Manheim, Crowston, and Chudoba, 2002). Most of the 
articles chosen for analysis came from journals in 
education research, while a few came from business 
journals. Although the articles selected for analysis 
ranged in publication date from 1996-2002, a majority 
of the articles (14 out of 21 chosen) were published 
between 2000 and 2002, as this seems to be the time at 
which online education emerged as a hot topic. A total 
of twenty-one articles were chosen for review based on 
the fact that they contained recommendations for how to 
design and teach an online course, offered “lessons 



learned” from real-life experience in teaching an online 
course, or presented issues to consider when teaching an 
online course. The second step in our analysis was to 
examine each article in search of common myths and 
metaphors embedded in the discourse on online 
education. Finally, after making a comprehensive list of 
the myths and metaphors found in these works, they 
were sorted in order to determine which myths and 
metaphors were the most prominent throughout these 
articles. 

ANALYSIS 

Myths 

Technological connectivity implies interaction. 

Interaction is one of the key components in any learning 
experience (Dewey, 1938; Vygotsky, 1978) and has 
become one of the most pervasive constructs in distance 
education. In fact, there exists the belief that online 
education provides an element of interaction that is 
absent in the traditional classroom. It has been 
suggested that unlike a traditional university, where 
students have difficulty finding time to meet with and to 
learn from one another, online education offers students 
the opportunity to interact whenever they have time 
(Shedletsky and Aitken, 2001). The technology used to 
teach online education is also said to facilitate and 
promote interaction through its ability to provide 
synchronous as well as asynchronous communication 
(Schrum, 1999) rendering face-to-face communication 
unnecessary (Liaw and Huang, 2000). The interactive 
component of technology is also believed to eliminate 
the time and space barriers between instructors and their 
physically distant students, thus recreating the classroom 
environment and allowing learners to engage in learning 
at their convenience with respect to place and time 
(Bernard, Rojo de Rubalcava, and St.Pierre, 2000). A 
further benefit of technology that appears in the 
literature is that the student can receive individual and 
immediate feedback and reinforcement from both 
instructors and peers (Raymond III, 2000). 

On-line courses are more effective because they 
embody a student-centered learning philosophy. 

Distance education researchers portray the traditional 
classroom setting as a teacher-centered, talking-head, 
passive-student model which is characterized by a 
boring lecturer who drones on while students sit idly by 
trying to absorb enough information so that they can 
regurgitate it for a test and forget it (Markel, 1999). 
Online education, on the other hand, is declared to be “a 
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more student-centered, collaborative, and egalitarian 
learning environment” (Weisenberg and Hutton, 1996). 
In this new paradigm, students become self-motivated 
managers of their own learning instead of passive 
bystanders, while instructors move from oracle, lecturer, 
and purveyor of knowledge to facilitator, guide, and 
mentor (Weisenberg and Hutton, 1996; Shedletsky and 
Aitken, 2001; Eastmond, 1996; Bernard, Rojo de 
Rubalcava, and St.Pierre, 2000; Raymond III, 2000; 
Murphy and Cifuentes, 200 1 ). In essence, it is believed 
that the online forum breaks down the teacher-student 
hierarchy (Weisenberg and Hutton, 1996). 

Any faculty member can teach an online course, any 
course can be taught online, and anyone can learn 
online. This myth starts with the belief that professors 
will be eager to participate in an online venture, and that 
being successful in this environment does not require 
any traits or skills that are different from teaching a 
traditional course. An additional notion is that the web 
is an appropriate medium for any type of course. This 
part of the myth views the web as a medium that enables 
the delivery of courses that were created within another 
framework (Carr-Chellman and Duchastel, 2001), and 
that all a professor needs to do is to make a few changes 
to his traditional course to prepare it for the online 
environment. In many cases, entire degree programs are 
being offered online, which inherently implies that any 
course can successfully be taught online. Online educa- 
tion is also seen as the remedy for the large masses of 
the population who for some reason or another cannot 
attend a traditional university. In fact, it has been stated 
that if technology continues its rapid growth, the need 
for student’s to be physically present in the classroom 
will be eliminated (Charp, 2000). This bold statement 
makes the assumption that the online learning 
environment is appropriate for everyone and that there 
are no technological, motivational, or cognitive/ 
intellective skill barriers that would prohibit entry into, 
or successful completion of, an online course or 
program. 

Faculty receive the same support and rewards for 
teaching an online course as they do for teaching a 
traditional course. The fate of a faculty member’s 
success in the online education domain relies to some 
degree on administrative decisions, even though these 
decisions may be made by people who have no expertise 
in computer pedagogy, scholarship, or general computer 
operation (Shedletsky and Aitken, 2001). This myth is 
based on the belief that administrators are aware of the 
time and effort that goes into converting a traditional 



course into an online course and are prepared to offer 
the technical support that may be required when 
designing and teaching an online course (Markel, 1999). 
It also presumes that course-load policies will be 
adjusted in recognition of the fact that an instructor 
teaching an online course spends significantly more time 
communicating with students on an individual level 
(Markel, 1999), thus making faculty workload signifi- 
cantly higher. The myth also implies that many faculty 
who are embarking on an online endeavor may be under 
the impression that they will retain the intellectual 
property rights to the courses they have created. They 
may also believe that they will be rewarded the same for 
teaching an online course as they will be for teaching a 
traditional course when it comes to issues such as tenure 
and promotion (Markel, 1999). 

The students and their instructor become one big 
happy family. The online environment is said to 
overcome isolation (Eastmond, 1996; Bernard, Rojo de 
Rubalcava, and St.Pierre, 2000), promote serendipitous 
encounters (Eastmond, 1996), and provide valuable 
intellectual exchanges profitable to all (Carr-Chellman 
and Duchastel, 2001). It is believed that online 
education transcends cultural, economic, and political 
systems thereby allowing us to increase our knowledge 
of one another and to recognize the similarities among 
the people of the world (Schrum, 1999; Anderson, 
2001). It is also alleged that this forum provides all 
students with a voice, thus everyone has the same 
opportunity to express their opinions and beliefs and no 
one person can dominate the conversation (Swan, 200 1 ). 

Metaphors 

Just-in-time learning. Primarily used in logistics, the 
concept of justAn^time processes originated from 
increasingly rapid modes of transportation and 
communication. The concept behind the idea was that 
items would arrive jrecisely at the time they were 
required for use or dispatch. Thus, in a learning context, 
this metaphor is used to describe the flexibility of online 
education. Just-in-time learning implies that the 
information can be communicated to students when and 
where they need it. In other words, students have access 
to the right information at the right time (Carr-Chellman 
and Duchastel, 200 1 ). In the traditional classroom, most 
instruction content quickly becomes “inert”, as it has 
little relevance to the life experiences of the learners 
(Gagne, Yekovich and Yekovich, 1993). However, it is 
believed that through the concept of “just-in-time” 
learning, which is made possible via the Internet, 
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learners can “download” their own knowledge as per 
their immediate requirements. The learner-objective 
environment promoted by the Internet makes learning 
resources and instructional activities available to the 
learner anywhere, anytime, thus allowing them to create 
links and search for knowledge that can interact with 
their own prior experiences (Shedletsky and Aitken, 
2001; Liaw and Huang, 2000; Carr-Chellman and 
Duchastel, 2001). 

Dual-mode institution. The word dual means of, or 
pertaining to, two. The word mode refers to a particular 
type or form of something, which in this context pertains 
to the way in which instruction is being offered. Thus 
the metaphor — dual-mode institution — is used to refer 
to those institutions that offer traditional residential 
programs as well as distance education programs 
(Camevale, 2000). The dual mode system is said to be 
characterized by an interlock of student choices 
exercised within a set of academic and organizational 
constraints (Guiton 1992). The metaphor also seems to 
imply that instituticais can undertake the responsibility 
of both traditional and on-line education with equal ease 
and effectiveness. 

Land-rush mentality. This metaphor creates the image 
that the race to enter into the online education domain is 
similar to the Oklahoma land rush of 1899, which has 
been called the most competitive event in history. Just 
as people dashed westward to stake their claim on a 
piece of land, both for-profit and non-profit institutions 
are scurrying to capture a piece of the online education 
market. Interestingly enough, the Okalahomaland rush 
meant disappointment to many, as the soil was not as 
rich as it had appeared and food and provisions were 
scarce. In fact, after months of starvation and being 
disillusioned, most returned to more civilized areas. 
Some say that the opening of the Indian lands in the 
Oklahoma Territory created as many problems as it 
solved. 

Sage on the Stage/Guide on the Side. This metaphor 
stems from the second myth mentioned above regarding 
the changing role of faculty. The word sage refers to an 
experienced or profoundly wise person, whereas a guide 
is someone who assists or supervises. Thus, in a 
traditional education setting, the instructor is referred to 
as a sage and the classroom serves as his stage. 
However, in the online domain, the instructor acts as 
more of a guide, assisting students in knowledge 
construction from the sidelines rather than projecting 
knowledge from center stagp. This metaphor therefore 



implies that on-line education requires a change in 
attitude/approach of the instructor in order to be 
effective (Coppola, Hiltz, and Rotter, 2002). 

Digital Diploma Mill Robert Reid, in his 1959 study 
of diploma mills for the American Council on 
Education, described the typical diploma mill as having 
no classrooms and faculty that are often untrained or 
nonexistent. Thus, the term “Digital Diploma Mill”, 
which was coined by David F. Noble, a professor at 
York University in Toronto, refers to the automation of 
higher education in which most teaching is done by 
machine, not faculty, and in which there are no 
classrooms. This metaphor conjures up a picture of an 
institution churning out commercialized, computer- 
based education created by faculty members who may or 
may not be involved in the dissemination of the 
courseware over the Internet. 

Corporatization. In the realm of online education, the 
term corporatization is used to “refer to the tendency of 
nonprofit organizations, such as universities, to operate 
more and more like businesses” (Molenda and Harris, 
2001). As government budgets are shrinking, nonprofit 
organizations are being forced to operate more and more 
like a for-profit business by generating new ways of 
supporting themselves in the marketplace. This 
metaphor implies an erosion of the basic values of 
academia such as academic freedom, autonomy over 
course content and pedagogical styles (Molenda and 
Harris, 2001). Instead, the term “corporatization” 
conjures up an image of a uniform and structured 
production-oriented environment that places value on 
productivity and contribution to the bottom-line. 

CONCLUSION AND FUTURE DIRECTIONS 

For fear of being left behind, maty universities are 
plunging into the online education waters without giving 
much thought to the pedagogical or economic costs that 
may follow. These institutions seem to think “everyone 
else is doing it, so there must be something to if’ — it is 
a competitive necessity. What is not known is the kind 
of impact this computer-mediated or computer-delivered 
education will have on the faculty and students. This 
paper has explored some of the myths and metaphors 
that seem to be embedded in the discourse on online 
education, in hopes of unearthing and carefully 
examining some of the commonly held implicit beliefs 
that surround this new phenomenon. These beliefs, 
whether accurate or not, will be responsible for molding 
the perceptions of those involved in online education. 
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which will in turn constitute educational practices for 
several years to come. Hence, it is critical that we 
question the myths and metaphors that pervade the 
online-education literature in an attempt to shed some 
light on the so-called truths that are being set forth. 

The next phase of our investigation will involve the 
interviewing of faculty members who have designed and 
taught a variety of infmnation systems courses online, 
and students who have taken such courses. The 
interview data, we are hopeful, will facilitate the 
discovery of additional symbolisms, and also allow a 
more definitive evaluation/elaboration of myths and 
metaphors discussed in the context of information 
systems education. 
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ABSTRACT 

At a time when demand for employees with IT skills is booming, there is a decline not only in the number of females 
entering this field of study in most western countries but also in the number of females taking computer courses 
[Nielsen, von Hellens, Wong, 2000]. Moreover, the gap in computer access and training for minorities is not 
declining [Hoffman and Novak, 2002]. However, data from Australia indicate that under-representation of women 
and minorities is somewhat less pronounced in programs in IT or Software Engineering than in programs in 
Computer Science or Computer Engineering [von Hellens and Nielson, 2000]. This paper seeks to determine 
whether there are similar differences in the state of Georgia in the U.S. 



INTRODUCTION 

According to a National Science Foundation-backed 
report from Carnegie Mellon University, the number of 
vacant IT positions is increasing by 25 percent per year 
[“How to Get More...,” 2000]. At a time when the law 
of supply and demand should be bringing females and 
minorities into high-paying IT jobs and companies want 
diversified personnel, the number of women and 
minorities in the workforce remains low [Gaudin, 1 999]. 
In fact in Canada the number of women graduating from 
IT programs has fallen by more than 50 percent in the 
last 15 years. While women make up 52 percent of the 
overall population, last year less than 25 percent of the 
computer technology graduates in Canada were female 
[Clow, 2002]. 

Data for Australia indicates that the under representation 
of women and minorities is somewhat less pronounced 
in programs in IT or Software Engineering than in pro- 
grams in Computer Science or Computer Engineering 



[von Hellens and Nielsen, 2001]. The aim of this 
research is to determine whether there are similar 
differences in the state of Georgia. 

REVIEW OF LITERATURE 

The evidence of a gender gap in those embracing 
technology is exhibited long before students arrive at 
universities. The American Association of University 
Women reports that boys are significantly more likely to 
take advanced computer classes in high school, to spend 
hours after school on a computer at home, and to enroll 
in computer camps [“Girls and Technology,” 1999]. 
While 56 percent of students enrolled in college are 
females, students who select a technology based major 
are predominately male. Some researchers feel that 
technology classes are presented in such a way, 
beginning as early as middle school, that females are 
discouraged from participating. In a study of high 
school students, females reported computing as boring, 
involving logic or math skills and little contact with 
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other people [Nielsen, et al., 2000] and were put off by 
the perceived heavy work load in computer classes 
[Clow, 2002]. In fact in a British survey cited by the 
Canadian Information Processing Society, it was 
reported that young wanen would rather be undertakers 
than work in IT [Clow, 2002]. Other research suggests 
that the masculine meaning that is ascribed to 
computing is offputting to females [Greenhill, von 
Hellens, Nielsen, Pringle, 1996]. 

The field of technology is not losing just the interest of 
females in middle and high schools. IT is rarely 
marketed to minorities. At an age when teenagers are 
very impressionable, minorities are almost never seen in 
print media advertisements. Few technology advertise- 
ments can be found in magazines aimed at the African 
American market, such as Emerge^ Essence^ Vibe, The 
Source, and Black Enterprise. The image of an IT 
professional is not marketed as “cool” to minorities. 
[Eglash, 2000]. 

When examining computer usage by race in the U.S., 
whites are significantly more likely to have a computer 
in their home and slightly more likely to have computer 
access at work. Studies show that even when adjusting 
for income and education, blacks and Hispanics have 
lower computer usage, computer access, and computer 
ownership. This gap is larger for high school graduates 
than for those without a high school degree, implying 
that education does not account for this gap. In recent 
years the percent of women owning computers has 
increased for whites but not for black women [Hoffman 
and Novak, 2002]. If minorities have limited exposure 
and access to computers, it is difficult for them to have 
the knowledge of and excitement for IT programs. 
Knowing what to expect in an IT degree program of 
study appears to be a problem that is not limited to 
minority populations. 

In surveying first year IT students at Griffith University 
in Australia, researchers discovered that both male and 
female students found introductory IT courses to be 
difficult and to differ greatly from the students’ 
expectations. This led to discouragement and disap- 
pointment [Nielsen, etal.,2000]. Mostof these students 
also had trouble envisioning the type of career they 
aspired to develop upon completion of the degree 
program [Greenhill, et al., 1996]. This may be in part 
due to the fact that IT is both a new and an evolving 
field [Nielson, von Hellens, and Wong, 2001]. The 
technical knowledge necessary to support the IT 



industry is increasing and expanding continually, as are 
career opportunities [von Hellens, Nielsen, Doyle, and 
Greenhill, 1999]. 

When comparing males and females, females were more 
likely to be looking for security and stability in a career 
and rated the ability to work from home as an important 
factor when considering a job. Females are also more 
interested in jobs that are people-oriented and require 
communication skills [Nielson, et al., 2001]. While 
these skills are needed in many jobs in IT (women are 
typically strong in data management and project 
management [Clow, 2002]), poor communication about 
this industry provides a false set of expectations and 
opportunities. Males on the other hand were interested 
in salaries and job satisfaction [Nielsen, von Hellens, 
Pringle, Greenhill, 1999]. 

IT employers seeking additional employees considered 
“soft” skills such as the ability to communicate, work in 
groups, adaptability, and self motivation more important 
than technical skills for new graduates. These skills are 
ones with which females are more likely to feel 
comfortable and are skills at which females are more 
likely to excel. It is easier for employers to teach 
technical skills in an industry in which technology is 
evolving. Yet females do not view these soft skills as 
skills needed for success in IT [von Hellens, Wong, and 
Orr, 2000]. 

It is interesting to note that in a study conducted in 
Australia it was found that successful women in IT had 
consciously chosen career over family. While many IT 
jobs provide the ability to work from home and many 
women would like to be able to take advantage of this, 
respondents stated that in an environment that is male- 
dominated, taking advantage of such perks as working 
from home or visibly placing family before career would 
impair their ability to advance and would result in the 
women not being taken seriously [Neilsen, et al., 2001 ]. 

Another striking difference in gender appears to be in 
the approach to group work. Females tend to view team 
projects as a collaborative effort and enjoy a sense of 
collectivism. On the other hand males shared infor- 
mation only on study related information and are more 
competitive [Nielsen, et al., 1999]. Male students also 
view females students as being less technically com- 
petent (which is not supported through testing) and tend 
to resist including females in projects or study groups 
[von Hellens, et al., 1999]. 
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Females in IT programs experienced gender-based 
discrimination. In a study conducted at Griffith Uni- 
versity in 1995, Asian women complained that their 
opinions were not valued and were largely ignored. Non- 
Asian women felt they were the focus of sexual 
harassment via loud boasting and uninvited e-mails. This 
group of females also complained of unwanted positive 
discrimination in the form of less strenuous evaluation by 
faculty [Nielsen, et al., 2001]. 

Females employed in IT positions believe that they 
experience gender-based pay discrimination. Statistics 
show that women make roughly 75 cents for every dollar 
their male counterparts earn. Some argue that the pay 
differentiation is a function of education and not gender. 
In a recent international survey for Deloitte & Touche of 
1500 IT professionals, 84 percent of the women surveyed 
believed that there was a plateau above which members of 
there gender could not advance. Statistics from the 
Department ofLabor support this belief Only 11 percent 
of all corporate executives are female [Alfe, 2002]. 

The aim of this research is to determine whether the 
differences in gender and ethnicity that are being 
observed in computer related degree programs can also 
seen in the public institutions of the state of Georgia. 
Specifically this research will examine breakdown by 
gender and ethnicity of baccalaureate students in IT, IS, 
and CS programs as a whole and by type of public 
institution. Public baccalaureate or above institutions of 
higher education in Georgia are classified as research 
universities, regional institutions, state universities, and 
state colleges (see Appendix A for a breakdown of 
Georgia institutions by type). Research universities have 
a statewide scope of influence in academic achievement, 
research, and public service and provide baccalaureate, 
masters, and doctoral degrees as well as a wide range of 
professional programs. Regional universities have a 
scope influenced by a specific region of the state as well 
as academic programs, research, and public service that 
are focused on regional need. State universities are 
influenced by the needs of an area of the state. State 
colleges are influenced by the needs identified within the 
local area in which they reside. By looking at programs 
and type of institutions, we can see if minorities and 
women are more likely to attend a specific type of 
university. 

RESEARCH QUESTIONS 

The goal of this research was to ascertain the breakdown 
by gender and ethnicity within public institutions offering 
baccalaureate or above degrees in Georgia, academic 



units within those institutions, and Information 
Technology and/or related programs of undergraduate 
students. Specifically gender and ethnicity was examined 
by college and degree programs that have a high level of 
technical content (i.e.. Computer Science, Information 
Systems, and Information Technology). 

Questions of interest were: 

1 . Is there a difference in gender mix between 
institutions, academic units within institutions, and 
IT-related programs within those institutions? 

2. Is there a difference in the gender mix and/or 
ethnicity between the institutions by classification? 

METHODOLOGY 

The provost/academic vice presidents at each selected 
Georgia public institution were contacted by e-mail and 
letter and provided a survey instrument. A follow-up 
contact was made via e-mail/telephone approximately two 
weeks after the initial contact. The respondent was asked 
if he/she would be willing to share the breakdown by 
gender and ethnicity of students at his/her institution. 
Specifically this research was seeking a breakdown by 
schools or colleges within the university as well as a 
breakdown within certain majors. 

To provide for consistency in data reporting. Spring 2002 
was specified in the data collection instrument as the 
enrollment data benchmark period. Since all Uni-versity 
of Georgia System institutions are on the semester system 
such a designation was possible. 

This study will serve as a pilot study. There are plans in 
place to follow-up this research to include institutions 
whose deans/directors participated in the Computer 
Research Association (CRA) Deans/Directors meetings 
and institutions affiliated with the newly formed Society 
for Information Technology Education (SITE). Future 
research will likely include discussions with focus groups 
composed of undergraduates who have chosen IT or an 
IT-related major at select Georgia institutions. 
Interaction with these groups may help provide 
information on how and why students select their field of 
study. 

RESULTS 

The state of Georgia has 20 public baccalaureate or 
higher degree granting institutions. Fifteen, or 75 
percent, responded to this survey. One research 
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institution, three state universities, and one state college 
failed to respond (see Appendix A). Two of the three 
state universities excluded from the results are 
historically black institutions. The inclusion of data 
from these two universities may have provided slightly 
different breakdowns of the student population by 
gender and ethnicity. 

With 75 percent of the institutions reporting, data 
indicates that female students outnumber male students 
(58 percent versus 42 percent). As Table 1 illustrates, 
the disparity is greater at state universities and colleges 
than at research or regional universities. 

While institutions of all types are comprised of more 
female than male undergraduates, this is not true of IT 



and IT-related degree programs. Only slightly more 
than 30 percent of the 873 undergraduates enrolled in IT 
degree programs statewide are female (see Table 2). 

Overall females make up a slightly greater percent of the 
IS student population (36 percent versus 31 percent). 
When examining undergraduate IS majors at research 
institutions, female students are almost equal in number 
to male students (see Table 3). 

The greatest gap in undergraduate population by gender 
in the programs canpared in this study is the Computer 
Science degree program. When comparing Computer 
Science students by gender, less than 20 percent of these 
degree majors are female (see Table 4). 



TABLE 1 

BREAKDOWN BY GENDER FOR TOTAL UNDERGRADUATE POPULATION 



Female 


Research 

56% 


Regional 

55% 


Universities 

60% 


Colleges 

66% 


Total 

58% 


Male 


44% 


45% 


40% 


34% 


42% 


Total Enrollment 


27,957 


17,768 


41,200 


3,626 


90,551 


TABLE 2 

BREAKDOWN BY GENDER FOR INFORMATION TECHNOLOGY MAJORS 




Female 


Research 

0% 


Regional 

20% 


Universities 

30% 


Colleges 

37% 


Total 

31% 


Male 


0% 


80% 


70% 


63% 


69% 


Total Enrollment 


0 


121 


380 


372 


873 


TABLE 3 

BREAKDOWN BY GENDER FOR INFORMATION SYSTEMS MAJORS 




Female 


Research 

47% 


Regional 

31% 


Universities 

38% 


Colleges 

0% 


Total 

36% 


Male 


53% 


69% 


62% 


0% 


64% 


Total Enrollment 


406 


297 


1,231 


0 


1,934 



TABLE 4 

BREAKDOWN BY GENDER FOR COMPUTER SCIENCE MAJORS 





Research 


Regional 


Universities 


Colleges 


Total 


Female 


12% 


19% 


24% 


0% 


19% 


Male 


88% 


81% 


76% 


0% 


81% 


Total Enrollment 


1,719 


218 


2,104 


0 


4,041 
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When breaking down the undergraduate population by 
ethnicity, 73 percent is white, 19 percent is black, and 8 
percent is of other race (see Table 5). These percentages 
do shift greatly when examining the different types of 
institutions in Georgia. Black students are a much 
smaller percent of the student body at research 
universities, “other” ethnic groups are a much higher 
percent. While the breakdown by ethnicity provides 
similar percentages at regional and state universities, 2 
historically black state universities did not respond to 
the survey. 

While statewide student population at Georgia 
institutions is 19 percent black, 32 percent of IT degree 
majors are black (see Table 6). When comparing the IT 
student population at the different types of institutions 
that have IT programs of study, the smallest percentage 
of black undergraduates is 25 percent at state colleges. 

Breakdown by ethnicity in IS degree programs yields 
varying results. OvQ-all, blacks and “other” races are 
over-represented when comparedto the population of all 
undergraduates (see Table 7). However, at research 
institutions ethnic lines in the IS program closely 
follows those of the general population at these 
institutions. 

When comparing the ethnic breakdown of students in 
the Computer Science degree program at institutions 



that offer this degree, there is once again a high 
concentration of blacks and “other” races (see Table 8). 
At research institutions, this degree program has a very 
high percentage of students from other ethnic 
backgrounds. 

It is interesting to note that black women and women of 
“other” ethnicity are over-represented in IT, IS, andCS 
degree programs. While 2 1 % of female undergraduates 
in Georgia are black, 33% of female CS majors, 37% of 
female IS majors, and 45% of female IT majors are 
black. Undergraduates of other ethnicity comprise 8% 
of the undergraduate population in the state of Georgia. 
Females of other ethnicity account for 15% of all 
undergraduate programs (see Table 9). In IT and IS 
degree programs, this select group of females makes up 
only 11% of the students, but in CS these women 
account for 20% of the CS majors. 

The differences in the makeup of male undergraduates 
by ethnicity in the IT, IS, andCS degree programs and 
the general male undergraduate population are less 
pronounced (see Table 10). In IT and IS, the percent of 
male students who are black is higher than the overall 
male population while in CS the percentage is slightly 
lower. In IS and CS, the percent of males who are of 
“other” race is greater than what is seen in all 
undergraduates in Georgia. 



TABLE 5 

BREAKDOWN BY ETHNICITY FOR TOTAL UNDERGRADUATE POPULATION 





Research 


Regional 


Universities 


Colleges 


Total 


Black 


6% 


26% 


23% 


36% 


19% 


White 


83% 


69% 


69% 


59% 


73% 


Other 


11% 


5% 


7% 


5% 


8% 


Total Enrollment 


27,957 


17,768 


41,200 


3,626 


90,551 



TABLE 6 

BREAKDOWN BY ETHNICITY FOR INFORMATION TECHNOLOGY MAJORS 





Research 


Regional 


Universities 


Colleges 


Total 


Black 


0% 


34% 


37% 


25% 


32% 


White 


0% 


61% 


51% 


67% 


59% 


Other 


0% 


5% 


12% 


8% 


9% 


Total Enrollment 


0 


121 


380 


372 


873 
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TABLE 7 

BREAKDOWN BY ETHNICITY FOR INFORMATION SYSTEMS MAJORS 





Research 


Regional 


Universities 


Colleges 


Total 


Black 


6% 


41% 


29% 


0% 


26% 


White 


76% 


53% 


60% 


0% 


62% 


Other 


17% 


6% 


11% 


0% 


12% 


Total Enrollment 


406 


297 


1,231 


0 


1,934 








TABLE 8 






BREAKDOWN BY 


ETHNICITY FOR COMPUTER SCIENCE MAJORS 






Research 


Regional 


Universities 


Colleges 


Total 


Black 


6% 


37% 


26% 


0% 


18% 


White 


68% 


52% 


60% 


0% 


63% 


Other 


27% 


11% 


14% 


0% 


19% 


Total Enrollment 


1,719 


218 


2,104 


0 


4,041 



TABLE 9 

BREAKDOWN BY ETHNICITY FOR WOMEN IN SELECTED DEGREE PROGRAMS 





IT Degree Programs 


IS Degree Programs 


CS Degree Programs 


All Programs 


Black 


45% 


37% 


33% 


21% 


White 


44% 


51% 


47% 


71% 


Other 


11% 


11% 


20% 


15% 


Total Enrollment 


274 


693 


758 


52,790 



TABLE 10 

BREAKDOWN BY ETHNICITY FOR MEN IN SELECTED DEGREE PROGRAMS 





IT Degree Programs 


IS Degree Programs 


CS Degree Programs 


All Programs 


Black 


25% 


20% 


15% 


16% 


White 


66% 


68% 


67% 


75% 


Other 


8% 


12% 


19% 


8% 


Total Enrollment 


599 


1,241 


3,283 


37,761 



CONCLUSIONS 

In Georgia there is evidence that the gender gap exists. 
While 58% of all undergraduates in the state are female, 
females account for a much smaller percent of those 
selecting IT, IS, or CS as their major. It appears that 
there is also a difference among students seeking these 
degrees when comparing ethnicity of these students 
versus the undergraduate population as a whole. IT, IS, 
and CS majors appear to be more diverse than the 
student population as a whole. This diversity spears to 



be greater at institutions that are not classified as 
research institutions. 

While this research supports the belief that a difference 
exists in the gender and ethnicity of students in technical 
degree programs, it does not address why. This study is 
a pilot study. Future research will likely include discus- 
sions with focus groups composed of undergraduates 
who have chosen IT or an IT-related major at select 
Georgia institutions. Interaction with these groups may 
help provide information on how and why students 
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select their field of study. However by itself this 
research may help guide graduate schools and 
companies that are seeking to increase diversity within 
their programs and corporations to recruit at institutions 
which offer a greater diversity in their student body. 
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APPENDIX A 

INSTITUTIONS RESPONDING TO SURVEY ON UNDERGRADUATE ENROLLMENT DATA 



Research Institutions: 

University of Georgia 
Georgia Institute of Technology 

Regional Institutions: 

Georgia Southern University 
Valdosta State University 

State Universities: 

Armstrong Atlantic University 
Augusta State University 
Columbus State University 
Fort Valley University 
Georgia College and State University 
Georgia Southwestern Univa*sity 
Kennesaw State University 
North Georgia University 
Southern Polytechnical University 
West Georgia University 

State Colleges: 

Macon State College 



INSTITUTIONS FAILING TO RESPOND TO 
SURVEY ON UNDERGRADUATE ENROLLMENT DATA 



Research Institutions: 

Georgia State University 

State Universities: 

Albany State University 

Clayton College and State University 

Savannah State University 

State Colleges: 

Dalton State College 
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CIS EXAM ADMINISTRATION 
OPEN VS. CLOSED NOTES 



Doris Duncan 

California State University^ Hayward 



ABSTRACT 

This paper summarizes findings of teaching six sections of the same college course in the same style with one 
exception. Section 1, 3, 4 and 5 had notes available for reference during the final exam whereas Sections 2 and 6 
did not. The author *s hypothesis that students in the 4 sections using notes would perform significantly better than 
the other sections on the exam was disproved 



INTRODUCTION 

Students often ask about use of reference aids during 
exams in most disciplines. These aids are most 
commonly notes they have taken, textbooks, handouts, 
or perhaps access to information stored on computers 
and/or the Internet. Although this practice may be 
common, the author was unable to locate any directly 
relevant research. 

METHODOLOGY 

Two hypotheses were formulated: 

HO: Use of reference notes will significantly 

improve performance of students on an exam. 

HI: Use of reference notes will not significanfly 

improve performance of students on an exam. 

To test these hypotheses the professor proceeded to 
teach six sections of the same course at the same 
university in the same way with the exception that 
Sections 1, 3, 4 and 5 were allowed to use reference 
notes during the final exam whereas Sections 2 and 6 
were not. 

The Subjects 

Subjects were enrolled in six sections of a computer 
information systems (CIS) course within the business 



school at a mid-sized state university in the USA. The 
full professor who taught all sections holds a Ph.D. in 
business with an emphasis in information systems and 
also holds professional certifications. 

Sections 1, 5 and 6 met 3 times a week for 70 minutes 
per session during the day. Section 3 met two times per 
week for 110 minutes per session in the afternoon. 
Sections 2 and 4 both met two times per week for 110 
minutes in the evening. 

All students in the six sections had the same reading 
assignments, the same two exams and the same five sets 
of homework, comprised of using spreadsheet, data 
base, internet, powerpoint and diagramming software. 
Students were expected to participate in class and the 
majority did. 

Students enrolled in the day sections (1 ,3> 5 and 6) were 
slightly younger than those in Sections 2 and 4 and 
tended to be full time students. Section 2 and 4 students 
mostly worked full time during the day and were 
enrolled in one or two evening classes. 

More females than males were enrolled in all six 
sections with the highest percentage of females being 
enrolled in the Section 4, an evening class. Section 5 
students attended class most regularly (83%) and 
Section 2 students attended class least often (69%). The 
demographic profile of students is summarized in Table 
1 . 
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TABLE 1 

DEMOGRAPHICS 



DEMOGRAPHIC 


SECTION 


SECTION 


SECTION 


SECTION 


SECTION 


SECTION 


ITEM 


1 


2 


3 


4 


5 


6 


NO. OF STUDENTS 
TAKING FINAL EXAM 


33 


28 


35 


28 


44 


34 


TIME OF MEETING 


Day 


Evening 


Day 


Evening 


Day 


Day 


FREQUENCY OF 
MEETINGS 


3 days per 
week 


2 evenings 
per week 


2 days per 
week 


2 evenings 
per week 


3 days per 
week 


3 days per week 


STUDENT MAJOR 


Business 


Business 


Business 


Business 


Business 


Business 
Administration- 
wide range 
of options 




Administra- 
tion, wide 
range of 
options 


Adm ini“ 
stration, 
wide range 
of options 


Administra- 
tion, wide 
range of 
options 


Administra 
-tion, wide 
range of 
options 


Administra- 

tion, 

wide range of 
options 


PORTION OF TIME 
WORKING 


Most part 
time 


Most full 
time 


Most part 
time 


Most full 
time 


Most part 
time 


Mostpart time 


RELATIVE AGE 


Younger 


Older 


Younger 


Older 


Younger 


Younger 


GENDER 


68.8% female 


62.1% 


55.6% 


71.4% 


56.8% 


61.8% 


DISTRIBUTION 


female 


female 


female 


female 


female 


ATTENDANCE: 
portion of class meetings 


74.4% 


69.3% 


77.7% 


75.0% 


83% 


76.6% 



attended 



The Exams 

All students took the same midterm and final exam. 
Both exams were comprised of 80 questions. All 
questions were multiple choice and had four items to 
choose from. Questions were based on a combination of 
lecture and reading assignments. The instructor made 
sure the material tested was covered in lectures and 
could be found in reading assignments as well. Students 
had at least 100 minutes to complete the final exam. 
They took approximately 12 minutes longer on average 
when using notes. Exams were machine scored. 

No section was allowed the use of notes during the 
midterm. Four of the six sections were allowed use of 
one page (8 by 1 1 inches) of handwritten notes for the 
final exam. Sections 2 and 6 were not allowed to use any 
notes. 

FINDINGS AND ANALYSIS 

T able 2 summarizes by section the averages of scores on 
exams and the number of students taking the exams. 
Interestingly, all six sections had higher average scores 
on the final exam than on the midterm. Although 
sections 2 and 6 did not use notes, they showed 
substantial improvement. Sections 1 and 3 showed the 
least amount of improvement even though they were 



both allowed to use notes. Thus the null hypothesis is 

supported. 

Possible Explanations 

• Since use of notes is optional, many students may 
choose not to use them and thus negate the potential 
impact. 

• Allowing notes may instill a false sense of confidence 
such that students do not prepare thoroughly for the 
exam. 

• Limiting reference notes to one page may be too 
similar to allowing no reference notes. 

• The final exam may be inherently easier than the 
midterm. 

• The instructor may have been more proficient in 
teaching sections 2 and 4 in the evening after having 
rehearsed in sections 1 and 3. (One must note, 
however, that sections 5 and 6, both day classes, 
showed the greatest improvement on the final.) 

• This may have been an anomaly. Two sections as the 
control group may be insufficient. 
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BEST COPY AVAILABLE 



TABLE 2 

SUMMARY OF EXAM RESULTS FOR 
CIS 3060 COMPUTER INFORMATION SYSTEMS IN MANAGEMENT 





Midterm 
# Students 


Midterm Average 
Score of 80 Questions 


Final Exam 
# Students 


Final Exam 
Average Score 
Of 80 Questions 


Change in 
Score 


SECTION 1 


33 


50 


33 


52.27 


+2.27 


SECTION 2 


32 


55 


28 


60.25 


+5.25 


SECTION 3 


35 


58.43 


35 


61.37 


+2.94 


SECTION 4 


29 


57.62 


28 


61.18 


+3.56 


SECTION 5 


43 


53.6 


44 


59.2 


+5.6 


SECTION 6 


36 


50.9 


34 


55.7 


+4.8 


AVG ALL SEC. 


34.67 


54.26 


33.83 


58.33 


+4.88 



CONCLUSION 

There was insignificant evidence to support the primary 
hypothesis that use of notes CIS exams enhances student 
performance. 

In the virtual classroom of the future, research such as 
this will be more challenging to conduct because of the 
difficulty in controlling reference materials available to 
students during exams. Exams as we know them today 
may cease to exist! 

POSSIBILITIES FOR FUTURE RESEARCH 

Several possibilities exist for additional research on 
ways to improve student perfonnance on exams: 

• A larger population of students taking this course 
should be tested to verify results; this should include 
at least one more section of students taking the final 
exam without use of notes. 



• Since use of notes was optional, the performance of 
students who choose to use notes should be compared 
with performance of those not using notes. 

• Other variables worthy of study include student 
demographics (portion male/female, etc.), attendance, 
time of day the class meets, and internet-based vs. 
face-to-face sections. 
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COURSE ASSESSMENT IN AN 
INTERDISCIPLINARY, INTEGRATIVE 
TEAM-TAUGHT E-COMMERCE COURSE 
IN THE COLLEGE OF BUSINESS 



Elaine Crable 
Xavier University 
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Xavier University 

Candace Gunnarsson 
Xavier University 

ABSTRACT 

This paper provides an assessment of an interdisciplinary, integrative, team-taught course in electronic commerce. 
The goal of the course was to first, integrate ecommerce within the functional area of marketing and information 
systems and then to integrate both these two functional areas of business. The result was an integrative team taught 
course that provided an innovative and holistic approach to strategic electronic commerce issues that exist in the 
current global market economy. The course design was a collaborative effort between a department of information 
systems and a department of marketing in the college of business at a Midwestern university. The assessment of the 
course covered: Student learning, students ' attitudes toward the integrative curriculum and faculty perception of 
success in the team teaching environment. 



INTRODUCTION 

In higher education today, there has been a proliferation 
of interdisciplinary teaching (Edwards, 1996; Gaff & 
Ratcliff, 1997; Klein, 1996). Although there are 
numerous reasons for this proliferation, one in particular 
is a paradigm shift in teaching style that has occurred 
within the last two decades. Academics in all disciplines 
have been moving from a lecture style pedagogy that 
promotes passive learning to an active learner focused, 
problem centered environment that incorporates active 
learning and collaboration. The natural outcome of this 
paradigm shift is interdisciplinary or integrative 
teaching. With the Internet as a learning tool by their 
side, it has become easier than ever for academics to 
organize course materials regardless of disciplinary 
boundaries. As more and more academics engage in the 
integration of different disciplines, a broadened context 



of ways to construct knowledge is established. (Newell, 
1998; DeZure, 2002). Furthermore, given the move 
toward globalization, both students and academics alike 
have began to crave connected learning and coherence 
in their curriculum (DeZure, 2002). 

In business schools in particular, the rapidly changing 
business environment has led many business schools to 
make the integration of curriculum a top agenda item. 
Furthermore, accrediting organizations such as, AACSB 
and ACBSP outwardly encourage schools of business to 
embrace interdisciplinary curriculum (Barber, Borin, 
Cerf & Swartz, 2002). Currently, these interdisciplinary 
course offerings can be found in all areas of the business 
school curriculum. Barber, et al, in their paper, “The 
Role of Marketing in an Iterative Business Curriculum,” 
cite three general models for integrating business 
curriculum. These models include 1) integration across 
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disciplines 2) integration of functional areas in business 
3) integration within a functional area. 

Although all three areas cited by Barber, et al. (2002) 
have been embraced, where integration appears to be the 
most prevalent is in new emerging disciplines. 
Therefore, when ecommerce exploded onto the scene in 
the 1990’s, many business schools quickly integrated 
ecommerce within the functional areas of either their 
marketing or information systems curriculum. It did 
not take long before faculty began to marry the 
ecommerce sides of marketing and information systems, 
whereby creating a new and exciting interdisciplinary 
and integrative approach to discovering and delivering 
ecommerce. 

This paper examines one such course. The course design 
was a collaborative effort between the department of 
information systems and the department of marketing 
and it was team taught by marketing and an information 
systems professor. This paper provides an assessment 
of the course within the following areas: student 
learning, students’ attitudes toward the integrative 
curriculum and faculty perception of success in the team 
teaching environment. 

METHODS 

At the beginning of the semester, students who were 
registered for the team taught electronic commerce 
course were asked to participate in a study to assess the 
effectiveness of the course they were about to undertake. 
Students that agreed to participate were instructed that 
they would be required to fill out a pre and post course 
evaluation questionnaire. Students were instructed that 
this was strictly voluntary and they were free to stop 
participating in this assessment anytime throughout the 
process. Students were told explicitly that failure to 
participate had absolutely no impact on their course 
grade. A total of thirty-nine students, 54% (2 1 ) male and 
46% (18) female, initially agreed to participate. The 
study finished with a total of 34 students, 56% (19) male 
and 44% (15) female. 

The course evaluation questionnaire focused on the 
following three major areas of interest: students’ 
anticipated comfort level with an integrative team 
teaching environment, students experience with 
technology and students expertise with electronic 
commerce. To assess faculty perception, no assessment 
tool was used. Instead, a qualitative approach was 
taken. Both faculty members agreed that at points in 



time throughout the semester they would journal their 
thoughts and feelings regarding the course and their 
perceptions regarding the effectiveness of the integrative 
team teaching experience. 

This research utilized a repeated measures design, 
whereby differences between the pre-course and post- 
course evaluation questionnaires were examined. The 
findings that follow cover three major areas: Student 
learning. Students’ attitudes toward team-teaching and 
faculty perception of success in the team teaching 
environment. 

FINDINGS 

Student Learning 

The goal of this electronic commerce course was that 
students would not only significantly increase their 
knowledge of electronic commerce, but that they would 
also increase their knowledge base when it came to 
certain technological applications. On both the pre and 
posts course evaluation questionnaires, students were 
asked to rate their level of experience regarding certain 
computer software and technology. Students’ experience 
regarding software technology stayed consistent, 
however, when it came to web based technology there 
was a significant increase in their knowledge base. All 
four questions regarding web based technology showed 
a statistically significant inaease in learned experience 
from before to after the course. Table 1 reports the 
finding for experience regarding computer software and 
technology before and after the course. 

Regarding their level of expertise in electronic 
commerce, students were asked to rate their level of 
expertise regarding electronic commerce foe the same 
nine questions both on the pre and post course 
evaluation questionnaires. Students in the beginning of 
the course had little, if any expertise, after the course 
their knowledge and expertise increased significantly for 
all ten questions. These findings are listed in Table 2. 

Based on the above findings, it is clear that the course 
did provide students with a significant knowledge base. 
Students have a greater understanding of web-based 
technology as well. 

Students’ Attitude Toward Team Teaching 

To assess students’ attitudes toward team teaching, 
students were asked to rate their comfort level working 
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TABLE 1 

EXPERIENCE WITH COMPUTER SOFTWARE AND TECHNOLOGY 

Scale 1 to 7 ( 1 - not comfortable to 7 - very comfortable) 

Rate your level of expertise with the following computer software and/or technology. 





Before 


After 


t 


P 


Microsoft Word 


6.16 


7.03 


-1.324 


.195 


Microsoft Excel 


5.13 


5.55 


-1.491 


.146 


HTML 


3.52 


3.06 


1.394 


.174 


Converting to HTML 


2.94 


2.84 


.346 


.731 


Web Browsers 


4.88 


5.19 


-.807 


.426 


Downloading Information from the Web 


5.34 


5.97 


-2.552 


.016* 


Purchasing Items Off the Web 


4.94 


5.50 


-2.119 


.042* 


Creating a Web Page 


3.29 


4.19 


-2.373 


.024* 


Maintaining a Web Page 


2.97 


4.06 


-2.884 


.007** 



^significant at the .05 level **significant at the .01 level 



TABLE 2 

STUDENTS EXPERTISE WITH ELECTRONIC COMMERCE 

Scale 1 to 7 (1 -not experienced, 7 -very experienced) 







Before 


After 


t 


P 


1. 


Preparing a business proposal fora new e-commerce initiative 


1.55 


4.55 


-9.242 


.000** 


2. 


Conducting a market/competitive analysis of an e-commerce 
industry 


1.66 


4.50 


-8.691 


.000** 


3. 


Determining how e-commerce products/services can be 
produced, distributed, and serviced 


2.26 


4.68 


-7.182 


.000** 


4. 


Developing a financial plan for an e-commerce business 


1.56 


3.75 


-7.661 


.000** 


5. 


Conducting web-based advertising, promotion, and other 
marketing activities 


1.91 


4.59 


-6.984 


.000** 


6. 


Developing interactive wd? sites that track purchases, process 
credit cards, etc. 


1.56 


3.63 


-5.532 


.000** 


7. 


Analyzing and selecting payment systems, web hosts, web 
development software, and other e-commerce resources 


1.52 


4.29 


-8.163 


.000** 


8. 


Understanding threats to e-commerce security and privacy 


2.28 


5.38 


-7.458 


.000** 


9. 


Recognizing the relationship between EC and other information 
systems 


1.84 


4.44 


-8.439 


.000** 


10. 


Analyzing the impact of EC on such social and legal issues as 


2.25 


5.09 


-8.308 


.000** 



censorship, privacy, international cultures, and equity among 
social classes 
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TABLE 3 

STUDENTS COMFORT LEVEL WORKING WITH 
TWO FACULTY MEMBERS FOR THE SAME COURSE 

Scale 1 to 7 (l=not comfortable, 7=very comfortable) 



Obtaining course feedback 
Receiving proper assessment 
Being graded feirly 
Receiving consistent information 



Pre 


Post 


t 


P 


5.94 


5.09 


2.655 


.012* 


5.94 


4.65 


3.712 


.001** 


5.97 


4.84 


3.000 


.005** 


5.69 


4.41 


3.772 


.001** 



♦significant at the .05 level **significant at the .01 level 



with two faculty members for the same course. They 
were also given the opportunity on the post test 
questionnaire to discuss their feelings toward the team 
teaching experience. A univariate repeated measures 
design was utilized to test for differences before and 
after the course. Statistically significant differences 
were found on all four questions pertaining to comfort 
level. Students comfort levels clearly decreased by the 
end of the course. Table 3 displays the four questions 
along with the means and significance levels. 

Faculty Perception of Success in the Team Teaching 
Environment 

Regarding faculty perceptions, both professors 
expressed initial concern regarding the particular 
challenges that they may face in the team teaching 
environment. Obstacles to team teaching cited by 
Campbell et al. (1997) were reiterated by both faculty 
members. These potential pitfalls included, teaching 
load, ego of faculty, professional sharing, coordination 
and communication. However, in spite of the potential 
pitfalls, each faculty member initially expressed great 
enthusiasm regarding their collaboration and believed 
the course would be successful. By the end of the 
semester, both faculty members still appeared positive. 
They both felt it was a positive learning experience for 
the students as well as each other. 



CONCLUSIONS 

This paper provides an assessment of an integrative 
team taught course in electronic commerce. The course 
was successful in providing an innovative and holistic 
approach to strategic electronic commerce issues that 
exists in the current global market economy. The course 
design was a successful collaborative effort. Student 
learning was significant in the areas of web based 
technology and electronic commerce. Students’ attitudes 
toward team-teaching overall was moderately 
comfortable, it did however decline from the beginning 
of the semester. 
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ABSTRACT 

Following the implementation of a strategy to overcome challenges associated with teaching a foundation MIS 
course to large cohorts of Business students, two surveys were conducted. The surveys were designed. to enable 
faculty to understand the attitudes, expectations, and computer experience of students, as well as to gauge the success 
of the new strategy. Analysis of the responses explores the association between students ' attitude to using computers, 
experience and self rated expertise with a variety of explications, and actual performance in course assessments. 



BACKGROUND 

Teaching an introductory MIS course involves 
overcoming challenges such as increased emphasis on 
flexible delivery, technology-experienced students, 
competition for technology and faculty resources, the 



dynamic nature of course content, and students’ 
expectations in teaching technology. Last year, in an 
effort to overcome these challenges, the University of 
Southern Queensland adopted a 4-pronged PATT 
strategy (Table 1). 



TABLE 1 

PATT STRATEGY 



Publisher 

Partnerships 

Flexible Assessment 

Styles 

Team 

Teaching 

Internet Technology 
Utilisation 



Collaboration with publishers can be leveraged to provide access to state-of-tho-art information and up-to-date 
textbooks. 

Employed by lecturer and teaching assistants (tutors) to target teaching at an appropriate level of students’ skills 
and experience (moran 1996). 

Continuity of course development, supports collaborative teaching, and offers complementary skills 

Used as an information dissemination tool, a discussion forum to encourage active participation and online 
assessment to increase accessibflity and improve learner control (nunan 1996). 



Source: Low, Genrich, Cater-Steel 200 1 
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In terms of student attendance, student performance and 
faculty satisfaction, the PATT strategy returned positive 
results. However, it was found that understanding the 
characteristics, attitudes and expectations of the diverse, 
technology-smart entry-level cohort is a critical factor in 
developing and delivering MIS courses to first year 
Business students. Bates (1994) argues that peoples’ 
attitudes towards IT and their access to it will be 
influenced by educational experiences and technologies 
utilised in work and leisure. 

METHOD 

The students involved in this study are all 
undergraduates enrolled in the Faculty of Business 
foundation MIS course. These students are working 
towards qualifications in a program of Accounting, 
Finance, Business, Information Technology (IT), or 
Economics. Within the program, students choose one 
major, or a combination of two major areas of study. 
The most popular majors offered by the Faculty of 
Business include Computer Software Development; IT 
Management; Marketing; Human Resources Manage- 
ment; e-Technologies; e-Commerce; Accounting; Law; 
and Economics and Resource Management. 

A two stage approach was taken to determine students’ 
attitudes towards the usefulness of computers while 
studying the foundation MIS course. Firstly, a web- 
based survey was conducted early in the semester to 
determine students’ perceptions about using computers. 
This questionnaire also collected data about the breadth 
and depth of students’ expertise with a range of 
computer appl ications, and how frequently they used the 
applications. 

The second stage of this research involved another web- 
based survey administered towards the end of the 
course. The focus of the second survey was on the 
students’ perceptions about the current foundation MIS 
course, and whether their expectations were met. In a 
course evaluation survey, students made positive 
qualitative comments about all aspects of the PATT 
strategy. However, further analysis of quantitative 
measures towards “attitude to using computers” revealed 
little correlation between survey indicators and attitudes. 

ANALYSIS 

In an effort to understand the students enrolled in the 
course, the following data were obtained: age, gender, 
whether the students had formal classes with computers 



at high school, type of school they attended, whether 
they owned a computer, when they first purchased a 
computer, the number of hours spent working on the 
computer, their self-rated skills in different applications 
of computers, and the frequency of use of the different 
applications, and their “attitude to using computers.” 

With a population size of 590 students enrolled in the 
Semester 1 offering of the foundation MIS course, 197 
students responded to the initial survey and 137 
responded to the second survey. 

“Attitude to using computers” is a modified version of 
a scale developed by Roberts (1991) to measure the 
attitudes of pre-service teachers towards computers. 
Some of the items in the original scale were considered 
inappropriate for business students, and hence the scale 
was reduced to eight items, with a resultant coefficient 
alpha of 0.86. 

The “attitude to using computers” measured on a five 
point scale (1 = strongly disagree, 5 = strongly agree) at 
the beginning of semester was 3.9183 (s.d. 0.7871) and 
at the end of semester was slightly less 3.7482 (s.d. 
0.6687). Possible explanations for this slight decline 
may be the timing of the surveys. The first survey was 
conducted before classes commenced and students may 
have been full of high expectations and may have been 
over-confident about their own ability and knowledge. 
The second survey was conducted just after Assignment 
3, and their performance on this assignment may have 
had a negative influence. The students’ initial view of 
computing may have been ‘blinkered’ by their 
experiences at high school, compared with the broader 
view they were exposed to within the course. 

Correlations between the items and the “attitude to using 
computers” were examined and found to be moderate 
for the hours of using a computer (r = .403, p= 0.000). 
The following section summarises the data collected in 
both surveys. 

TABLE 2 



USAGE 


HOURS 




Hours/week spent on 


Survey 


Survey 


Computer 


1 


2 


0-2 hours 


8.9% 


3.7% 


3-6 hours 


23.2% 


17.6% 


7-10 hours 


16.3% 


16.2% 


More than 10 hours 


51.6% 


61.8% 



316 



Proceedings of the 17^ Annual Conference of the International Academy for Information Management 



There has been a shift upwards in the amount of time the 
students spend on the computer during the semester as 
shown in Table 2. This could be attributed the demands 
placed on the students in their enrolled courses. 

As shown in Table 3, the students rated their expertise 
with email (4.28) and word processing (4.05) very high 
ahead of all surveyed applications. The correlations with 
“attitude to using computers” are significant, but too 
small to be meaningful. Even though the students use e- 
mail, and word processors frequently, and they rate 
themselves highly with these applications, they are not 
good indicators of a positive attitude to using computers. 
Further studies need to be undertaken to identify factors 
that would be better indicators of positive attitude to 
using computers. 

As expected, the packages that the students consider 
themselves to be best with are the packages they use the 
most. Once again, the coffelations with “attitude to 
using computers” are significant, but low. None of the 
items tested correlated highly with “attitude to using 
computers.” An interesting point is that the students use 
the word processes, e-mail and Web browsers the most; 



perhaps they view the computer as a “tool for 
communication.” 

FINDINGS 

In analysing “attitude to using computers,” it was 
discovered that age, gender, computer education at high 
school, ownership of a computer, length of usage of 
computers, and perceived expectations all have little 
bearing on the respondents’ attitude to using computers. 
These findings could be attributed to the fact that those 
students who perceive themselves to have high skills 
and who frequently use computer applications do not 
necessarily have the most positive attitudes to using 
computers. Contrary to expectations, high school 
computer education, and ownership of a computer at 
home were not significantly associated with positive 
attitudes about computers. These findings challenge the 
‘nerd’ stereotype that positive attitudes are associated 
with young, male, students. The results obtained are 
consistent with the findings of an earlier study which 
focused on students in an Education Faculty. Table 4 
explains the possible reasons for the lack of correlation. 



TABLE 3 

SELF-RATED EXPERTISE AND FREQUENCY OF USE 
CORRELATED WITH “ATTITUDE TO USING COMPUTERS” 



Self-rated expertise 



Frequency of Use 





Mean 


Std. Dev. 


R (sig. 1-tail) 


Mean 


Std. Dev. 


R (sig. 1-tail) 


E-mail 


4.28 


.898 


.363 (.000) 


2.30 


1.128 


.272 (.000) 


Word Processor 


4.05 


.776 


.282 (.000) 


2.59 


1.112 


.252 (.000) 


Web Browser 


3.90 


1.177 


.445 (.000) 


2.20 


1.153 


.324 (.000) 


Spreadsheet 


3.63 


.879 


.306 (.000) 


1.99 


1.140 


.302 (.000) 


Presentation Software 


3.33 


1.050 


.318 (.000) 


1.08 


.807 


.246 (.000) 


Mail Merge 


3.30 


1.383 


.371 (.000) 


1.17 


.969 


.248 (.000) 


Database 


2.98 


1.003 


.253 (.000) 


1.49 


1.133 


.275 (.000) 


Newsgroups 


2.93 


1.605 


.372 (.000) 


1.03 


.988 


.211 (.001) 


Desktop Publishing 


2.67 


1.333 


.262 (.000) 


.83 


.798 


.244 (.000) 


Chat 


2.65 


1.742 


.300 (.000) 


.89 


1.135 


.131 (.033) 


Accounting 


2.55 


1.549 


.211 (.001) 


1.32 


1.479 


.183 (.005) 


Drawing 


2.05 


1.384 


.212 (.001) 


.55 


.780 


.198 (.003) 


File Transfer Protocol 


1.78 


1.608 


.239 (.000) 


.41 


.687 


.210 (.002) 


Gopher 


1.10 


1.300 


.187 (.004) 


.27 


.546 


.169 (.009) 
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TABLE 4 

LACK OF CORRELATION BETWEEN INDICATORS AND ‘‘ATTITUDE TO USING COMPUTERS” 



Indicator Reason 

Formal Classes at High School It was discovered that almost all respondents have had some formal classes with 

computers. 



Ownership of a Computer The majority of respondents own a computer. 



Length of Computer Most the respondents have had computers in their homes for many years — 

Ownership belonging to parents or older siblings. 



Length of Usage Most respondents spend time on computers, regardless of their attitude towards 

them, either for study or work. Therefore, time spend on the computer was 
found to not be optional. It is the accepted norm that Foundation MIS students 
have to use applications such as word processors, and e-mail for study and 
work, and do so frequently, so skill in using these and how frequently they are 
use explains little about their attitude to using computers. 



Further analysis compared the students’ perceived 
expertise with actual performance in course 
assessments. There is a tendency for students with 
perceived high skill levels to become overconfident and 
consequently perform poorly possibly because they 
think they have an adequate grasp of the foundation MIS 
content. Initial analysis of the survey confirmed this 
tendency: students with high perceptions of skills did 
not gain high assessment results. 

FURTHER RESEARCH 

The survey is being improved and further smaller 
surveys conducted at periodic intervals throughout the 
semester. This is aimed to allow the measure of 
student’s attitude more frequently to determine the 
pattern of fluctuations in attitude. Focus groups will be 
conducted to determine the underlying causes of attitude 
variation. 

As well as providing the course team with an insight 
into curriculum design for future offerings of this 
introductory MIS course, this study has also assisted in 
further examining the efficacy of implementing the 
PATT strategy and its impact on student attitudes 
towards computing. Many students have strong 
prejudices, positive or negative, which impact on their 
performance. This study could assist bothpre-tertiary 
and tertiary educators in aligning their curriculum, to 



facilitate the smooth transition from pre- to tertiary-level 

MIS studies. 
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ABSTRACT 

This research explores the role cf CASE tools in information technology (IT) education. What principles of systems 
development do CASE tools promote? Tools may allow students to more efficiently design and develop effective 
information systems (IS) but does automating design result in more ^fective learning? Or have IT faculty enabled 
students to learn bad habits more efficiently? The research identifies factors that influence the adoption and use 
CASE tools in academic settings. In a recent inquiry, IT academics shared CASE tool experiences and presented 
evaluations. Analyses of their responses lead to preliminary interviews with employers about the pervasiveness of 
CASE tools and the rationale behind their use. Insights from the comments and theory from research in IS 
development and diffusion of innovation theory are presented to offer a theoretical model of IT academic CASE tool 
adoption and use. The research will help develop future research venues dealing with the choice of tools and the 
complexity of these decisions. Do the selection criteria used by faculty match those of practitioners? What obstacles 
affect the decisions to adopt CASE in general or certain took specifically, in terms of both time and cost, both 
initially and over its lifetime? Results can provide insight into the ability of the IT discipline to advance theory and 
practice in systems analysis and design. 



INTRODUCTION 

Computer-Aided Systems Engineering (CASE) tools are 
software used by systems developers to support the 
diagramming, analysis, and debugging of data and 
process models used for application development. These 
tools, in use since the 1970s, were often only in large 
firms that could afford to deploy them due to their 
expense, training requirements, and learning curve for 
IT staff. More recently less-expensive packages have 
increased the pervasiveness of their use. Academia has 
mirrored practice by offering instruction in CASE tools 
for systems analysis and design. However the artificial 
classroom setting imposes limitations on how well 
students can acquire expertise not only in using 
particular tools but also more importantly in 
understanding modeling concepts. Academic 



experiences take into account many different aspects of 
systems development by offering a variety of courses. 
Each program designs offerings based on their emphasis 
and philosophy; some programs have more technical 
approaches, Miile others accent business analysis. The 
delivery reflects these themes and is operationalized in 
resource choices. The lack of IT standardization results 
in many tool choices in practice and many options for 
academics to teach modeling. 

Questions arise about differences between academic and 
business settings. Academics’ concern is their ability to 
teach the appropriate methodologies and techniques. In 
industry the ability to improve efficiency and increase 
quality over the total development and maintenance of 
applications takes precedence. These differences and 
others influence the decision of what tools to use and 

319 



Proceedings of the 17^ Annual Conference of the International Academy for Information Management 



o 

ERIC 



3G2 



BEST COPY AVAILABLE 



inevitably impact the teaching of modeling. Classroom 
adoption of CASE tools is influenced by factors similar 
to adoption of innovation factors but can differ in how 
they are operationalized due to different objectives for 
using CASE tools by academia and practice. In 
summary, context matters with regard to CASE tool 
adoption. 

The paper examines the systems development principles 
that IT academics espouse in using CASE tools and how 
academics choose diagramming tools to teach process 
and data modeling. Discussion does not include other 
CASE features such as debugging and code generation. 
This paper discusses, the value of using CASE tools 
across the IT curriculum and not merely in an isolated 
course, the criteria applied in curriculum decisions using 
innovation adoption theory as a base, and preliminary 
results from a survey of IT academics that can shed light 
on the criteria that faculty use for making tool choices. 
A model for CASE tool adoption and use in informatics 
education is proposed along with a plan for further 
research. 

WHAT INFLUENCES THE 
ADOPTION OF CASE TOOLS? 

How does the use of CASE tools help students 
understand how to model information requirements and 
business processes? IT faculty teach students how to 
identify, scope, and solve information problems in 
organizations. One of the most difficult skills to develop 
in systems analysis is to organize user-articulated 
information needs. IT faculty also teach methodologies, 
techniques and tools. For instance, modeling helps 
students learn to analyze and identify problems in an 
existing system, define and set its boundaries, and 
decompose the system’s components in order to 
minimize its complexity. Examples include data flow 
diagrams (DFDs) and entity relationship diagrams 
(ERDs). Other underlying principles taught in IT 
programs are methods and standards, teamwork, project 
management, user involvement, and documentation 
among others. CASE tools offer a repository of 
documentation that is accessible by a project team of 
users and developers. They help enforce standardization 
of name conventions and allow balancing checks. 

The argument for using CASE tools is based on systems 
development concepts. Students follow methodological 
approaches from systems theory, as well as techniques, 
which hopefully leads to improved learning, and 
relevant experience modelling practice. Theories should 



guide teaching practices, including methodological 
approaches and techniques, which in turn influence 
adoption and use of tools. An underlying theme for 
using CASE tools is to walk the talk. Systems ^ecialists 
who cannot apply technology to their own craft need to 
reconsider arguments for espousing it to users. 
Therefore, practice grounded in theory should guide and 
influence adoption and use of tools. 

CASE tool adoption and use in industry has received 
attention from IS researchers examining motivations for 
adoption and use both quantitatively to identify 
predicting factors and qualitatively to develop insight 
into social and political forces (Fichman, 2000; Fichman 
and Kremerer, 1999; livari, 1996; Aaen, et al, 1992, 
Orlikowski 1993, Rojas et al, 2001). Notable are 
research findings showing that CASE acquisition does 
not insure adoption or use (Fichman, 2000). To measure 
the effectiveness of the tool, it is important to measure 
its adoption and use as well as its acquisition. CASE is 
an example of second order innovation, or type -two 
technology affecting organization routine with potential 
barriers to adoption and use. 

IT faculty in their role as CASE tool adopters reflect 
different priorities than practitioners. Academics focus 
on tools’ ability to support teadiing appropriate 
methodologies and techniques. Classroom adoption of 
CASE tools is influenced by factors similar to practice 
but due to different objectives. The context influences 
CASE tool adoption. The objectives of the adoption that 
define successful adoption and use must be defined 
distinctly. Practitioners focus on tools’ ability to 
improve efficiency and increase the quality of software 
development and maintenance. Aaen (1996) examined 
industry CASE acquisitions and use by measuring 
perceived productivity and quality effects. To measure 
successful adoption and use of tools for teaching and 
learning, Aaen’s measures need to be changed to fit the 
academic objectives of perceived understanding, 
comprehension or retention. The efficacy of CASE tools 
may be qualitatively different for pedagogical purposes. 
Context appears to matter in consideration of factors 
influencing CASE adoption and use; therefore, measures 
of effectiveness should be reconsidered, as should the 
operationalization of predicting factors. 

An important consideration for CASE curricula 
decisions is minimizing the complexity for students and 
faculty. In practice allows long-term learning. Despite 
periods of high turnover, most firms QxpQci to retain FT 
staff longer than a few months. In fact, they often make 



320 



Proceedings of the 17^ Annual Conference of the International Academy for Information Management 



investments in their training anticipating years of 
productivity. In academia the artificial time constraint of 
a school term impedes long-term learning. If faculty 
could integrate one or a minimal number of tools over 
several courses, students become more familiar and gain 
more experience. This could lower the learning curve. 
As a result, students could graduate with increased skill 
level and an understanding of the power that vendors 
purport CASE offers. 

Faculty of highly cohesive IT programs are more likely 
to make tool choices collaboratively with the intent of 
fewer tools. Despite other reasons, the rationale for an 
integrated tool across courses is often financial. 
Different tools for different courses and models confuse 
students and prevent them from acquiring depth of 
understanding. Lab support necessitates centralized 
approval, thereby forcing faculty to discuss merits and 
concerns in making choices. 

The rationale for an integrated tool across courses is 
often financial. Cost considerations affect faculty, 
student, and administrative funds and time presenting 
obstacles for choosing some tools. Some tools have 1-2 
hour tutorials that can provide enough expertise because 
the tool is not robust. Other choices such as Oracle 
necessitate multiple weeklong classes to reinforce 
understanding to exploit the tool’s power. The quality of 
faculty training impacts the quality of student training. 
Most packages come shrink-wrapped with a textbook. 
The practice of purchasing used texts confounds the 
purchase of bundled software in a course. Students 
without computers of their own and those trying to save 
on book expenses may find some products prohibitive 
making lab facilities a necessity. Maintenance requires 
IT services support and/or release time for a faculty 
coordinator. This entails not only technical support, 
server allocation and security, but also administration of 
files by someone who understands the nature of class 
assignments, their objectives, and the interrelationships 
of courses. Technical support staff cannot be expected 
to have that type of breadth. It necessitates faculty time 
and involvement. 

RESPONSE RESULTS 

In response to a request on IS World, 39 IT academics 
shared diagramming products used and comments about 
the efficacy of these products. Table 1 contains the list 
and Web sites, if available, of those mentioned. The list 
only includes those products \\here a sample of IT 
academics has indicated their use. The last entry links to 



a more exhaustive set of products. Forty percent were 
using Visio. Although comments were sparse about why 
Visio was chosen, what was more interesting was 
comments about why other products were not chosen. 
Many of the concerns reflected issues discussed above 
about how to integrate CASE tools in the curriculum. 
Table I contains a brief description of the issues that 
surfaced. 



TABLE 1 

IT ACADEMIC CASE TOOL CHOICE ISSUES 



Issue 


Description 


Faculty 


Time and funding for seminar/ 


training 


class attendance 


support 


Recognition or course reduction 
for self-taught training 


Financial 


License cost 


Costs 


Lab/network configuration costs 
Cost to students 
Maintenance support 


Multi-use tool 


Most tools excelled at one function 
Integrated use desired e.g. one tool 
for multiple course covering 
multiple modeling techniques 


Ease of 


Students ability to use the tool and 


Learning 


focus on methods rather than on 
the tool itself 


Maintenance 


Technical lab/hetwork support 
Faculty methodology repositories 


Textbook 

Publishers’ 


Ability to package tool with 


collaboration 


textbooks 



DISCUSSION 

The investigation provoked a number of issues. Recent 
emphasis on assessment of IT programs (Noll and 
Wilkins, 2002) has forced faculty to look in the mirror 
and to evaluate their program’s performance. Feedback 
from students, employers, and alumni often focus on 
software training. The choice of tools can directly 
impact how students grasp basic systems development 
principles and concepts. The implementation of that 
choice through adoption and use varies widely. What are 
the factors that e^qjlain the variance and then impact the 
effectiveness of its use for students? The Aaen, et al 
(1996) model and research in innovation adoption and 
IT adoption (Davis et al., 1989; Moore and Benbasat, 
1991; Fichman, 2000) proves relevant. Taking into 
account the contextual differences that may affect CASE 
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adoption in IS curriculum, the model is customized to 
examine the importance of overall objectives that 
influence academic adoption criteria and behavior. 
Factors include those from IS implementation, diffusion 
of innovation, and IS adoption research. 

Factors from IS Implementation Research: 
Participation, Management Support, Training, and 
Expectation Realism 

Participation looks at the degree to which users 
participate in its acquisition class (Hartwick and Barki 
1994). Faculty not participating in the selection process 
may circumvent or undermine mandatory use in a 
curriculum or in a specific course reflecting beliefs that 
the tool is inapprq)riate or too complex. The 
investigation found departments where adoption 
preceded faculty hiring. Classroom use in some of those 
cases was erratic and highly inconsistent. The 
construct’s conceptualization is consistent with livari 
(1996), investigating the number of faculty involved in 
an adoption decision. 

Top management support for financial and technical 
resources may predetermine adoption decisions. livari 
included the management of the technical sujport 
department. Failure to update a tool’s versions, 
hardware, and operating system could easily impede 
ongoing use or adoption by potentially new adopters. 
The financial cost associated with tools was repeatedly 
raised from the respondents and was even as an inhibitor 
to choose desired alternatives. Larger commercial CASE 
tools require a commitment not just to increased 
licensing fees, but also to dedicated servers, adequate 
client PCs with operating systems and memory suitable 
to their demands. 

Faculty training is an important factor. Faculty trained 
on the software tool or who have access to quality 
materials are more likely to increase adoption and use. 
Training helps faculty understand the full potential of 
the tool and how best to exploit its capabilities. 
Investment in training also has practical implications; 
the time it would take to learn another tool would 
present potentially significant switching cost for faculty 
and departments. livari included in-house training, 
vendor courses, and self-study material. The factor was 
measured in terms of adequacy and quality. For the 
academic community, additional material may pertain to 
and support their process of instructing students, as well 
as understanding the tools themselves. Course material 
supplied through instructor’s manuals for the software. 



Web sites hosted by vendors, user groups through 
universities or professional organizations, and 
integration with the textbook represents a variety of 
training. 

Expectation realism measures the difference between the 
expected perceived benefit of adopting and using a 
CASE tool and the resulting perceived performance. 
Unrealistic expectations can cause success to be elusive 
(Ginzberg, 1981). Faculty expectations about how easily 
students can learn and use a tool, how well it 
communicates the concepts, and how effective the 
overall experience is may diminish quickly if the tool is 
overly complex or difficult to use. All the more reason 
to involve faculty in the choice decision. livari used five 
items for this factor which primarily measured 
performance outputs such as impact on speed of 
development, functional quality of new applications, 
productivity, cost and quality of development, and the 
cost of maintenance. The measures for a curriculum 
setting require slight modification to focus on 
pedagogical performance. For instance, speed of 
development time will be influenced as much if not 
more by the amount of experience students have had 
with modeling techniques as by the CASE tool. 

Factors from Diffusion of Innovation Research: 
Complexity, Compatibility, and Relative Advantage 

The livari (1996) model drew from the technology 
acceptance model (TAM) with measures of complexity 
representing perceived ease of use (Davis, et al, 1989). 
He altered the focus from the user interface quality to 
whether “underlying concepts, limitations and assump- 
tions of the tool are easy to understand.” While this 
conceptual turn proves insightful and useful, additional 
dimensions of complexity need examination that goes 
beyond perceptions and deals with some specific 
problems that CASE presents to students. CASE tools 
present a paradigm shift for developers by moving 
beyond a set of stand-alone functions to exploiting a set 
of integrated tools capturing interrelationship among 
design components. For example, some tools allow 
developers to place restrictions on the data field of an 
ERD that a DFD process can modify. These 
dependencies, implemented as design rules, constrain 
designers by enforcing methodology that prevents 
mistakes and functional and technical problems. As 
students modify their designs, students are not always 
aware of the interrelationship do to the way they are 
technically implemented. This can confound iterative 
development as students become frustrated and halt the 
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design process prematurely. The integrative nature of 
CASE can pose problems for students working in teams. 

Compatibility is the degree to which a new technology 
is perceived to be consistent with values, needs and 
experiences of adopters (Moore and Benbasat,). CASE 
needs to be compatible on two levels: technical 
infrastructure and course content. Updating or 
purchasing computers, operation system software or 
servers that increases the cost and complexity beyond 
the organization’sability could negatively influence the 
adoption decision. Therefore faculty would favor tools 
relatively compatible with the existing environment. A 
tool’s embedded techniques and methodologies may 
restrict alternate methods. For example, Oracle Designer 
employs traditional structured design but not object 
oriented methods. 

Relative advantage measures the degree to which faculty 
perceive an innovation as better than its precursor. The 
set of responses indicated many departments were 
changing or had done so recently. The degree to which 
they believe that a new tool is better relative to the old 
should predict the likelihood of use, for example, the 
function to balance DFD levels. 



Factors from IS Adoption Research: Voluntariness, 
Trialability, Demonstration of Results, and Ease of 
Distribution 

Since the use of some tools is mandated in 
organizations, voluntariness asks if the use is required or 
voluntary. However, mandatory use of a tool in a 
department may not always be correlated with high 
degrees of use, since the faculty member may limit the 
degree to which they use the tool. Trialability is the 
degree to which users can experiment with an 
innovation before its adoption. Most faculty become 
aware of new teaching material through publisher 
samples or vendor trial versions. Results demonstiat- 
ability combines observability and communicability to 
yield the tangibility of the results. Faculty rely on 
observing students and on their feedback about their lab 
and home experience. Favorable results will therefore, 
be made apparent to the instructor in these two ways. 
Ease of distribution facilitates students’ repeated access, 
unlike one time as in many adoption studies. For 
instance, textbooks offer software at little additional 
cost. 



FIGURE 1 

MODEL OF FACTORS AFFECTING ACADEMIC CHOICE OF CASE TOOLS 
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CONCLUSION AND IMPLICATIONS FOR 
FUTURE RESEARCH 

The research presented in these proceedings is a work- 
in-progress. A refined model is anticipated by the time 
of the conference. A more exhaustive literature search 
will probably encourage restructuring the model 
indicating causal and precursor relationships. For the 
short term, measures of the constructs will be 
determined to validate the model and pilot those. Model 
testing will proceed using a sample of faculty solicited 
from ISWorld who have recently been involved in a 
CASE acquisition or review of existing tools. 

The refinement of constructs and relationships as a 
result of the data and its analysis will give us a closer 
look at issues. A major premise is the difference 
between academic and practitioner settings. Future 
research could expand on livari’s model for industry 
data collection and compare that analysis to the findings 
to confirm the differences and similarities. Although the 
time frame is one of the major limitations for academics, 
the nature of learning may be more similar than 
expected. If academics indeed are mirroring practice in 
their choices and the research indicates that the 
differences question mirroring the choices, researchers 
need to explore what type of criteria are important for 
educatorsto consider so students assimilate successfully 
in the practitioner world. A more panted question is 
does it matter? Many students might be employees at 
firms that have no CASE tools. Others may use CASE 
tools upon graduation but they may not be ones as 
robust as they used in an academic program. Does that 
experience hone their modeling and problem solving 
skills offering an indirect benefit to industry? Or are the 
classroom exercises a waste of resources? 

A neglected stakeholder is the student. Although they do 
are not directly responsible for the acquisition decision, 
their indirect input is vital and may need more weight 
than some faculty allow. One avenue to explore student 
input will be a pilot in an undergraduate systems 
analysis class. Students will respond to skill assessment 
and attitude measures before instruction in a CASE tool. 
After three months of adoption and use, the attitude 
measures will be repeated along with open-ended 
responses concerning the learning experience. Seven 
months after initial instruction the students will repeat 
the attitude measures and respond to perceived 
performance success measures. This pilot will help test 
the constructs. IT educators need to review the 
objectives for using CASE tools and question their 



efficacy. In addition to role of CASE in enhancing 
student’s placement and having long-temi benefit to 
their career, does a more robust tool achieve oie goal to 
help students understand the complexity of systems and 
their interrelationships? 

Feedback for vendors to improve CASE tools that meet 
the needs of both industry and education is another 
outcome of this stream of research. IT academics and 
practitioners depend on vendors for the tools but they 
depend on educators to train future users in practice and 
depend on researchers to question issues and advance 
knowledge. The latter can offer improvements that the 
vendors bring to market. CASE tool vendors have a 
stake in the research outcomes. Besides helping them to 
determine target markets, research about how students 
learn modeling can help the design of their products and 
how to deploy training. Other aspects of the model as 
proposed and how it evolves can lead to improved 
CASE tools that benefit all stakeholders. 
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APPENDIX A 

CASE TOOLS USED BY IT PROGRAMS 



Product 


Web site 


ABCFlowcharter 




ADONIS 


www.boc-eu.com 


ASCENT 




AXIOM 


www.stgcase.com/casetools/axiomsys.html 


ConceptBase 


www-i5.informatik.rwth-aachen.de/CBdoc/ 


CoolGen 


www3.ca.com/Solutions/Product.asp?ID=256 


Decamerone 




DBMain 




Edge Diagrammer 


www,pacestar.com 


Embarcadero ER-Studio 


www.embarcadero.org/products/erstudio/index.asp 


Embarcadero Describer 


www.embarcadero.org/products/describe/index.asp 


ERD 




ERWin 


www3.ca.com/Solutions/Product.asp?ID=260 


InterProcs 


www.euridis.fbk.eur.nl/?view=Interprocs 


MetaCase/MetaEdit+ 


www.metacase.com/ 


Omnigraffle 


www.omnigroup.com/applications/omnigraffle/ 


Oracle Designer 


www.oracle.com/ip/developyids/index.html7designer.html 


Rational Rose 


www.rational.com/products/rose/index.jsp 


Relationship Diagrammer 


Relationship Diagrammer 


Select SSADM 


www.aonix.com/content/products/select/select_ssadm.htm 


SmartDraw 


www.smartdraw.com/specials/diagram.asp?id=2541 


Systems Architect 


www.systemsarchitect.com 


Together Control Center 


www.togethersoft.com/ 


Visible Analyst 


www.visible.com/Products/Analyst/overview.html 


Visio 


www.msdnaa.com 


MetaEditH- 


www.metacase.com/ 


Graphical tools link pages 


www.methods-tools.com/tools^modeling.html 
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ABSTRACT 

The Information Systems discipline has a long history of applying model curricula to guide the particular course 
offerings of academic institutions. This paper provides a synopsis of the final report of IS 2002 — The Model 
Curriculum and Guidelines for Undergraduate Degree Programs in Information Systems. The last major update, 
IS '97, was completed several years ago. The need for the update is warranted due to the advent of the Internet, the 
changes in student computing literacy, and the recent Information Systems accreditation movement. The curriculum 
updates follow the recommendations of three collaborating organizations: AIS, ACM and AITP. After presenting an 
introduction and discussion of the need for a curriculum update, the paper describes the key principles and guiding 
assumptions about the Information Systems field. Next, the exit characteristics of IS graduates are presented. This 
motivates a discussion and presentation of the scope of the current curriculum update. The paper concludes by 
presenting the course architecture, sequence and high-level catalog descriptions. 



INTRODUCTION 

The Model Curriculum and Guidelines for Under- 
graduate Degree Programs in Information Systems (IS 



2002) is the latest report on model curriculum work in 
Information Systems. Theprevious report was presented 
in 1997 (IS’97). The first infoimation systems (IS) 
curriculum effort took place in 1972 (ACM’72, 



326 



Proceedings of the 17^ Annual Conference of the International Academyfor Information Management 



0^0 
kJ C O 



BEST COPY AVAILABLE 



Ashenhurst, 1972) and then again in 1982 (ACM’ 82, 
Nunamaker et al., 1982) by the Association for 
Computing Machinery (ACM). Other organizations, 
including AIS (Association for Information Systems), 
AITP (Association of Information Technology 
Professionals), and IFIP (International Federation for 
Information) have published IS model curricula. IS ’97 
was the first major collaboration of the three key 
organizations with a worldwide membersh^ in 
information systems: ACM, AIS and AITP. IS 2002 is 
the second collaborative effort between ACM, AIS and 
AITP. IS 2002 is also part of the Computing Curricula 
2001 project (CC2001), which is a joint undertaking of 
the Computer Society of the Institute for Electrical and 
Electronic Engineers (lEEE-CS) and the Association for 
Computing Machinery (ACM) to develop curriculum 
guidelines for undergraduate programs in computing. 

In addition to the cooperating organizations ACM, AIS 
and AITP, IS 2002 had by the end of October 2002 been 
endorsed by the Decision Sciences Institute (DSI), the 
International Association fa* Computer Information 
Systems (lACIS), the International Academy for 
Information Management (lAIM), and the Society for 
Information Management (SIM). Endorsements by 
AITP Special Interest Groq) for Education (EDSIG), 
ACM Special Interest Group on Management 
Information Systems (SIGMB), and INFORMS 
Information Systems Society (INFORMS-IS) were at 
that time pending. 

Although ACM, AIS and AITP are worldwide 
organizations, IS 2002 does not represent a universal 
global IS curriculum. It does not seek to harmonize the 
curriculum to meet the requirements of different 
educational systems around the world. The model 
curriculum for undergraduate degree programs in 
information systems is based on the typical degree 
structure in USA and Canadian universities. The IS 
model curriculum can serve as a useful reference by 
curriculum designers and developers outside North 
America in designing and developing information 
systems degree programs as its predecessors have done 
over the past decades. 

The most recent undergraduate curriculummodel, IS’97 
(Davis et al., 1997) was circulated in draft form in 1994 
(Gorgone et al., 1994; IS’95, see Longenecker et al., 
1994) and 1995 (Couger et al., 1995) and finalized in 
1996. Members of the Joint Task Force presented drafts 
of IS’97 at numerous conferences from 1994 to 1996 
and received significant feedback that substantially 



strengthened the report. It has been approximately seven 
years since the currait model curriculum has been 
updated since much of the work was actually completed 
between 1994 and 1995. Since 1997, the task force 
members have collected survey data to better understand 
the changes that have occurred in the Information 
Systems discipline. This paper provides a progress 
report on the updating of IS’97. 

The next section is a brief discussion of the need to 
update IS’97. This is followed by key principles and 
guiding assumptions about the IS profession that helped 
to shape the curriculum design and evolution. The exit 
characteristics of IS graduates are presented next. The 
scope of the curriculum update is then discussed so that 
programs transitioningfrom IS’97 to IS2002 will better 
track the model’s evolution. This is followed by a brief 
presentation of course architecture and intended course 
sequence. Finally, the paper concludes by providing 
course prerequisites and high-level course catalog 
descriptions of the IS 2002 model curriculum. 

NEED FOR CURRICULUM UPDATE 

Since the last revision of the undergraduate curriculum 
guidelines, three major factors have purred the need to 
reexamine and update the existing standard. In 
particular, the advent of the Internet, the changes in 
student computing literacy, and the information 
accreditation movement all motivate the need to update 
the curriculum. In this section, we briefly review each of 
these motivating factors. 

During the development of IS’97, the true importance of 
Web and Internet systems development and related 
concepts such as Web-based distributed architectures, 
Internet protocols, and IP-based internetworking was not 
yet foreseen. Alttiough it was known that the impact of 
these then novel concepts could be enormous, their real 
effect was at the time unrealized and not fully 
understood. In the intervening years, the Internet has 
grown to become a major aspect of all IS environments. 

Over the past decade, there has been a significant 
change in basic computer literacy of incoming 
university students. In the past, very few students 
entered a university having significant skills in using a 
desktop computer, with even fewer students owning or 
having easy access to a computer. Today, with the 
advent of the Internet revolution and low cost PCs, most 
students entering a university today have at least a 
modest level of computer literacy. 
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There has been interest in the accreditation of programs 
in Information Systems since the accreditation of 
programs in computer science was begun in mid- 1 980s. 
The work on IS ’97 with its support from the major IS 
professional societies provided the much needed catalyst 
for IS accreditation to move forward. With the support 
of the National Science Foundation, the Criteria for the 
Accreditation of Programs in Information Systems have 
been developed with IS ’97 serving as the basis of the IS 
curriculum criteria. ABET is the agency with 
responsibility for accrediting all programs in computing, 
engineering and technology (Gorgone & Lidtke, 2002). 
The Computing Accreditation Commission (CAC) has 
responsibility for accrediting Computer Science and 
Information Systems programs. The first IS 
accreditation visit of an IS program was completed 
during Fall 2001. 

KEY GUIDING PRINCIPLES 

The key principles that guided the IS 2002 development 
effort were as follows. 

1 . The model curriculum should represent a consensus 
from IS community. 

2. The model curriculum should be designed to help IS 
faculty produce competent and confident entry level 
graduates well suited for workplace responsibilities. 

3 . The model curriculum should not be prescriptive; by 
using the model curriculum guidelines, faculty can 
design their own courses. 

4. The model curriculum should be based on sound 
educational methodologies and make appropriate 
recommendations for consideration by IS faculty. 

5. The model curriculum should be flexible and 
adaptable to most IS programs. 

GUIDING ASSUMPTIONS ABOUT 
THE IS PROFESSION 

In conceptualizing the role of information systems in the 
future and the requirements for IS curricula, it seems 
apparent that several elements have been (ACM ’72, 
ACM’82, IS’95, IS’97, and ISCC’99) and remain 
important and characteristic of the discipline. These 
characteristics evolve around four major areas that 
permeate all aspects of the IS profession and therefore 
must be carefully integrated into any IS curriculum: 



1 . IS professionals must have a broad business and real 
world perspective. Students must therefore 
understand that: 

• IS enable the success of organizations; 

• IS span and integrate all organizational levels and 
business functions; and 

• IS are increasingly of strategic significance 
because of the scope of organizational systems 
involved and the role systems play in enabling 
organizational strategy. 

2. IS professionals must have strong analytical and 
critical thinking skills. Students must therefore: 

• Be problem solvers and critical thinkers; 

• Use systems concepts for understanding and 
framing problems; 

• Be capable in applying both traditional and newly 
learned concepts and skills; and 

• Understand that a system consists of people, 
procedures, hardware, software and data. 

3. IS professionals must have strong ethical principles 
and interpersonal communication and team skills. 
Students must understand that 

• IS requires the application of professional codes of 
conduct; 

• IS requires collaboration as well as successful 
individual effort; 

• IS design and management demands excellent 
communication (oral, written, and listening); and 

• IS requires persistence, curiosity, creativity, risk 
taking and a tolerance of these abilities in others. 

4. IS professionals must design and implement 
information technology solutions that enhance 
organizational performance. Students must therefore: 

• Possess skills in understanding and modeling 
organizational processes and data, defining and 
implementing technical solutions, managing 
projects, and integrating systems; 
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• Be fluent in techniques for acquiring, converting, 
transmitting, and storing information; and 

• Focus on the application of information 
technology in helping people and organizations 
achieve their goals. 

EXIT CHARACTERISTICS OF IS GRADUATES 

The graduates of an undergraduate IS program should be 
equipped to function in an entry-level position and 
should have a basis for continued career growth (Lee, 
Trauth and Farwell 1995; Landry et al 2000). Figure 1 
presents a high-level categorization of the exit 
characteristics that emphasizes the central role of 
Technology-Enabled Business Development at the 
intersection of the four major areas that were identified 
in the initial assumptions about the IS profession. 
Although not presented here, the full curriculum report 
divides the main categories further into subcategories 
and presents concrete, practical representative examples 
of the exit characteristics in each subcategory. The 
overarching objective for IS professionals is to enable 
organizations to utilize information technology to 



achieve their strategic obj ectives with a customer service 
orientation. 

SCOPE OF CURRICULUM UPDATE 

Recently a survey of computing faculty in the United 
States was conducted to ascertain information in two 
areas. The first was to determine their current view of the 
appropriate depth of mastery for each of the elements in 
the IS ’97 body of knowledge. The second was to gather 
similar information for key skill areas identified within 
IS’97. Some of the observations based on this research 
have been published (Longenecker et al., 1 999; Landry et 
al., 2000). The primary conclusions can be summarized 
as: 

1 . IS analysts have specific skills at approximately IS ’97 
skill depth level 3 (the ability to USE know-ledge) in 
areas of Interpersonal and Team Skills, Business 
Knowledge, Organizational Process Development 
(including IS Systems Analysis and Design), Project 
Management, Database, Software Development, Web 
Programming, and Systems Integration. 



FIGURE 1 

FUNDAMENTAL CATEGORIES OF EXIT CHARACTERISTICS OF IS GRADUATES 
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2. Skills identified in IS’97 as Exit Curriculum Areas 
match expectations of the computing industry as 
well as IS faculty. 

3. Skill areas produced by programs of Information 
Technology match expectations of Information 
Systems faculty. 

4. The model courses of IS’97 are acceptable to both 
IS and IT faculty. Interestingly, both Computer 
Science and Software Engineering faculty also feel 
that IS’97 courses are relevant. 

When analyzing the survey data and the IS’97 
curriculum in summer of 2001, the missing element in 
the curriculum was a course focusing on Internet-based 
commerce. At most universities, this course has been a 
popular required course for several years and so there 
was a clear discrepancy in the existing model curriculum 
and what was being operationalized in most universities. 
Thus, a new course was added to the model curriculum, 
IS 2002. 2-Electronic Business Strategy, Architecture 
and Design, to address this limitation. Without restruc- 
turing other aspects of the model curriculum, the 
addition of a new Internet-based commerce course to the 
model curriculum would result in curriculum of eleven 
required courses. There was a desire by the committee 
to limit model curriculum to a target of ten courses, or 
30 semester units, which therefore would result in the 
elimination of one of the existing IS’97 courses. This 
decision was driven by recently approved accreditation 
standards and by the practical credit hour constraints 
realized by many IS programs. 

IS’97 had a prerequisite course, \^"91 .VO-Knowledge 
Work Software Tool Kit, that assumed students had 
elementary exposure to a suite of software tools useful 
for knowledge workers (spreadsheets, databases, 
presentation graphics, database retrieval, statistics, word 
processing, and electronic mail). It was also assumed 
that students could gain this knowledge through aformal 
course or through self-study modules. Beyond this 
course, IS’97 had a required course, 91 .2-Personal 

Productivity with IS Technology, that focused on 
improving student skills in using packaged software, in 
both individual and group work, by designing and 
developing solutions. Given the need to limit the IS 
2002 curriculum to ten courses and the rapid and 
significant improvement in the general computing 
literacy of entering students, IS’97.P0 andIS’97.2 were 
merged into a single prerequisite course, IS 2002.P0- 
Personal Productivity with IS Technology. 



Beyond the addition of the Internet-based commerce 
course, IS 2002.2, and the merging of IS97.P0 and 
IS’97.2 into a single prerequisite course (or self-study 
modules, see below for more details), the remaining 
courses were retained with appropriate updating of the 
course “Scope” and “T opic” descriptions. In most cases, 
this resulted in the addition of Internet centric-content 
and more contemporary terminology and concepts. For 
all but one, course names remained unchanged. 
YiowQ\QV,lS'91 .9-PhysicalDesign and Implementation 
with a Programming Environment was changed to IS 
2002.9-Physical Design and Implementation in 
Emerging Environments. This change was motivated by 
the continual evolution in rapid application development 
and programming environments. A mapping of the 
courses and a summary of the changes is represented in 
Table 1. 

COURSE ARCHITECTURE AND SEQUENCE 

The IS 2002 curriculum assumes that students have a 
prerequisite knowledge of desktop computing. 
Specifically, it is assumed that students have an 
elementary exposure to a suite of software applications 
useful for knowledge workers such as word processing, 
spreadsheets. Email, and Internet browsing. In addition, 
students are assumed to also have knowledge and skill 
of IS technology to be a successful knowledge worker as 
described in the prerequisite course IS 
2002. ^0-Personal Productivity with IS Technology. 

Figure 2 shows the course architecture and sequence of 
courses within IS 2002, including the prerequisite course 
IS 2002. PO. The structure is a “suggested” architecture 
and sequence with the appreciation that each 
university’s situation is somewhat unique. In any event, 
this architecture allows the entire program to be 
completed within a scope of two years. This model will 
therefore fit within the broader curricula constraints of 
most business schools. For IS programs housed outside 
business, great flexibility in the sequence can be 
employed. 

COURSE DESCRIPTIONS 

In this section, we conclude by providing high-level 
course catalog descr^tions for IS 2002. These include 
the prerequisite Personal Productivity with IS 
Technology (IS 2002.P0) course plus the ten courses 
within the Model Curriculum. Presented in Table 2 
below is the title, prerequisite and catalog description 
for each of the courses. 
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TABLE 1 

MAPPING OF IS 2002 COURSES TO IS’97 COURSES 



IS 2002 Courses 






IS’97 Courses 








I S’ 97 P0-Know4 edge Work Software Tool 


IS 2002.P0 - Personal Productivi^ with 
IS Technology 




MERGING 


K.lt 

IS’ 97.2 -Personal Productivi^ with IS 








— Technology 


IS 2002.1 -Fundamentals of 
Information Systems 




UPDATE 


IS’ 97.1 -Fundamentals of Information 
Systems 










IS 2002.2 - Electronic Business 
Strategy, Architecture and Design 




NEW 


NO EXISTING COURSE 


IS 2002.3 - Infonnation Systems 
Theory and Practice 




UPDATE 


IS’ 97.3 - Information Systems Theory and 
Practice 








IS 2002.4 - Information Technology 
Hardware and ^sterns Software 




UPDATE 


I S’ 97 .4 - Informati on T echnol o gy Hardware 
and ^sterns Software 








IS 2002.5 - Programming, Data, File, 
and Object Structures 




UPDATE 


IS’97. 5 - Programming, Data, File, and 
Obj ect Structures 








IS 2002.6 - Networks and 
Telecommunication 




UPDATE 


IS’ 97.6 - Networks and Telecommunication 








IS 2002.7 -Analysis and Logical 
Design 




UPDATE 


IS’ 97.7 - Analysis and Logical Design 








IS 2002.8 -Physical Design and 
In^rlementation with DBMS 




UPDATE 


IS’ 97.8 - Physical Design and 






In^l ementati on v^th DEI^S 


IS 2002.9 -Physical Design and 
In^lementation in Emerging 
Environments 




UPDATE 


IS’ 97.9 - Physical Design and 






In^rlementati on with a Programming 
Environment 


IS 2002.10 -Project Management and 
Practice 




UPDATE 


IS’ 97. 1 0 - Proj ect Management and Practice 
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FIGURE 2 

IS 2002 COURSE ARCHITECTURE AND SEQUENCE 




SUMMARY AND CONCLUSION 

This paper presented a status report on the current 
updates to the information system model curriculum for 
2002. Discussed were the key motivations behind the 
need to update IS’ 97, key principles and guiding 
assumptions about the IS profession that helped to shape 
the curriculum design, and the exit characteristics of IS 
graduates. The scope of the curriculum update was 
discussed so that programs transitioning from IS’97 to 
IS 2002 will better understand the model’s evolution. 
This was followed by a brief presentation of the course 
architecture and intended course sequence. Finally, the 
paper concluded by providing high-level course catalog 
descriptions of the IS 2002 model curriculum. 



It could be easily argued that the curriculum updating 
process is too slow and not responsive to the pace of 
change encountered in practice. In the future, as 
information technologies continue to evolve and change 
at an increasingly rapid pace, the need for even more 
timely updates is clear. Arguably, the largest motivation 
for the current curriculum update was the advent of the 
Internet. This global architecture will provide a 
necessary foundation to ease the updating process and 
facilitate broader involvement by the IS academic and 
professional community. Just as the Internet has 
fundamentally changed business, we too feel that this 
medium can be used to fundamentally change the 
curriculum design process. As we move forward, we 
urge the reader to embrace this opportunity to shape the 
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TABLE 2 

IS 2002 COURSE NUMBER, TITLE, AND CATALOG DESCRIPTION 



Course Number & Title 
(prerequisite) 


Catalog Description 


IS 2002,P0-PersonaI Productivity with 
IS Technology (basic word piDcessing, 
spreadsheets, e-mail, and Web browsing) 


Students with minimal skills will learn to enhance their personal productivity 
and problem solving skills by applying information technologies to problem 
situations and by designing and using small information systems fo r individuals 
and group s. 


IS 2002. 1-Fundamentals of 
Information Systems (IS 2002. PO) 


Systems theory, quality, decision-making and the organizational role of 
information systems are introduced. Information technology including 
computing and telecommunications systems are stressed. Concepts of 
organization and information system growth and re-engineering are introduced. 


IS 2002.2-£lectronic Business Strategy, 
Architecture and Design (IS 2002. 1 ) 


The course focuses on the linkage between organizational strategy and 
networked information technologies to implement a rich variety of business 
models in the national and global contexts connecting individuals, businesses, 
governments, and other organizations to each other. The course provides an 
introduction to e-business strategy and the development and architecture of e- 
business solutions and their components. 


IS 2002.3-Information Systems Theory 
and Practice (IS 2002.1) 


Students who have con structed personal info rmation systems will be exposed to 
the theory of the IS discipline. Application of these theories to the success of 
organizations and to the roles of management, users and IS professionals are 
presented . 


IS 2002.4-Information Technology 
Hardware and System Software (IS 
2002.1) 


Principles and application of computer hardware and software will be presented 
through lecture, installation, configuration, and operations experiences. 


IS 2002 .5-Pro gram ming, D ata, File 
and Object Structures (IS 2002.1) 


This course presents object oriented and procedural software engineering 
methodologies in data definition and measurement, abstract data type 
construction and use in developing screen editors, reports and other IS 
applications using data structures including indexed files. 


IS 2002.6-Networks and 
Telecommunications (IS 2002.4) 


Students will gain in-depth experience ofnetworking and telecommunications 
fundamentals including LANs, MANs, and WANs. Data communication and 
telecommunication concepts, models, standards, and protocols will be studied. 
Installation, configuration, system s integration and m anagem ent of infrastmeture 
technologies will be practiced. 


IS 2002.7-Analysis and Logical Design 
(IS 2002.1) 


Students with information technology skills will learn to analyze and design 
information systems. Students will practice project management during team 
oriented analysis and design ofa departmental level system. 


IS 2002.8-Physical Design and 
Implementation with DBMS (IS 2002.5 
and IS 2002.7) 


Students who have completed the analysis and logical design course will 
develop a detailed physical design and implementation based on a logical design 
utilizing a DBMS. The course integrates intensive project work with relevant 
concepts. 


IS 2002.9-Physical Design and 
Implementation in Emerging 
Environments (IS 2002.2 and IS 2002.8) 


Students who have completed the analysis and logical design course will extend 
their knowledge by implementing an information system using a contemporary 
developmentenviron ment capable of in teracting with a local or a remote DBMS. 
Teams will use project management principles to implement an information 
system. 


IS 2002.10-Project Management and 
Practice (IS 2002.7) 


Advanced IS majors operating as a high-performance team will engage in and 
complete the design and implementation of a significant information system. 
Project management, management of the IS function and systems integration 
will be components of the project experience. 
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IS academic discipline so that we can collectively be 
more responsive to the changingcurricular needs of both 
academia and industry. 

ENDNOTE 

1 . An earlier version of this paper was published in the 
Proceedings of the 2002 American Conference on 
Information Systems, Dallas, Texas. 
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